[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250114 02:52:41.031842 425 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.106.126:45803
I20250114 02:52:41.033157 425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:52:41.034062 425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:52:41.053167 432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:52:41.053995 431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:52:41.055429 434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:52:41.056321 425 server_base.cc:1034] running on GCE node
I20250114 02:52:42.184291 425 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 02:52:42.184543 425 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 02:52:42.184731 425 hybrid_clock.cc:648] HybridClock initialized: now 1736823162184710 us; error 0 us; skew 500 ppm
I20250114 02:52:42.185544 425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:52:42.192387 425 webserver.cc:458] Webserver started at http://127.0.106.126:37773/ using document root <none> and password file <none>
I20250114 02:52:42.193204 425 fs_manager.cc:362] Metadata directory not provided
I20250114 02:52:42.193413 425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:52:42.193819 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:52:42.199834 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/master-0-root/instance:
uuid: "7eb855ea5ab8482d8facc17c41aea2ac"
format_stamp: "Formatted at 2025-01-14 02:52:42 on dist-test-slave-k78q"
I20250114 02:52:42.207808 425 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250114 02:52:42.212775 440 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:52:42.213927 425 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250114 02:52:42.214237 425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/master-0-root
uuid: "7eb855ea5ab8482d8facc17c41aea2ac"
format_stamp: "Formatted at 2025-01-14 02:52:42 on dist-test-slave-k78q"
I20250114 02:52:42.214594 425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:52:42.284842 425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:52:42.286285 425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:52:42.286734 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:52:42.359468 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.126:45803
I20250114 02:52:42.359561 491 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.126:45803 every 8 connection(s)
I20250114 02:52:42.367276 492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:42.390070 492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac: Bootstrap starting.
I20250114 02:52:42.395807 492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:42.397343 492 log.cc:826] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac: Log is configured to *not* fsync() on all Append() calls
I20250114 02:52:42.401762 492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac: No bootstrap required, opened a new log
I20250114 02:52:42.424353 492 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb855ea5ab8482d8facc17c41aea2ac" member_type: VOTER }
I20250114 02:52:42.424917 492 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:42.425171 492 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb855ea5ab8482d8facc17c41aea2ac, State: Initialized, Role: FOLLOWER
I20250114 02:52:42.425987 492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb855ea5ab8482d8facc17c41aea2ac" member_type: VOTER }
I20250114 02:52:42.426461 492 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 02:52:42.426738 492 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 02:52:42.427039 492 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:42.430967 492 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb855ea5ab8482d8facc17c41aea2ac" member_type: VOTER }
I20250114 02:52:42.431732 492 leader_election.cc:304] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eb855ea5ab8482d8facc17c41aea2ac; no voters:
I20250114 02:52:42.433844 492 leader_election.cc:290] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 02:52:42.434244 495 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:52:42.437424 495 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [term 1 LEADER]: Becoming Leader. State: Replica: 7eb855ea5ab8482d8facc17c41aea2ac, State: Running, Role: LEADER
I20250114 02:52:42.438277 495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb855ea5ab8482d8facc17c41aea2ac" member_type: VOTER }
I20250114 02:52:42.439855 492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [sys.catalog]: configured and running, proceeding with master startup.
I20250114 02:52:42.451928 496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb855ea5ab8482d8facc17c41aea2ac. Latest consensus state: current_term: 1 leader_uuid: "7eb855ea5ab8482d8facc17c41aea2ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb855ea5ab8482d8facc17c41aea2ac" member_type: VOTER } }
I20250114 02:52:42.451917 497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eb855ea5ab8482d8facc17c41aea2ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb855ea5ab8482d8facc17c41aea2ac" member_type: VOTER } }
I20250114 02:52:42.452848 496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [sys.catalog]: This master's current role is: LEADER
I20250114 02:52:42.453020 497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [sys.catalog]: This master's current role is: LEADER
I20250114 02:52:42.457995 504 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 02:52:42.470120 504 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 02:52:42.473264 425 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 02:52:42.486491 504 catalog_manager.cc:1348] Generated new cluster ID: 13640c82e9e647e8a028d1a74afec1e7
I20250114 02:52:42.486796 504 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 02:52:42.533324 504 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 02:52:42.535552 504 catalog_manager.cc:1505] Loading token signing keys...
I20250114 02:52:42.551607 504 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac: Generated new TSK 0
I20250114 02:52:42.552592 504 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 02:52:42.604380 425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:52:42.614064 514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:52:42.615257 515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:52:42.620469 517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:52:42.621711 425 server_base.cc:1034] running on GCE node
I20250114 02:52:42.622601 425 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 02:52:42.622848 425 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 02:52:42.623030 425 hybrid_clock.cc:648] HybridClock initialized: now 1736823162623015 us; error 0 us; skew 500 ppm
I20250114 02:52:42.623764 425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:52:42.627744 425 webserver.cc:458] Webserver started at http://127.0.106.65:38461/ using document root <none> and password file <none>
I20250114 02:52:42.628470 425 fs_manager.cc:362] Metadata directory not provided
I20250114 02:52:42.628713 425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:52:42.629056 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:52:42.630581 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-0-root/instance:
uuid: "ef76faab133b4cbf96d9e7610be21b62"
format_stamp: "Formatted at 2025-01-14 02:52:42 on dist-test-slave-k78q"
I20250114 02:52:42.637487 425 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250114 02:52:42.642959 522 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:52:42.643764 425 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250114 02:52:42.644083 425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-0-root
uuid: "ef76faab133b4cbf96d9e7610be21b62"
format_stamp: "Formatted at 2025-01-14 02:52:42 on dist-test-slave-k78q"
I20250114 02:52:42.644515 425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:52:42.677700 425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:52:42.679594 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:52:42.682869 425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:52:42.688297 425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:52:42.688642 425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:52:42.688970 425 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:52:42.689193 425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 02:52:42.795348 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.65:36183
I20250114 02:52:42.795485 584 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.65:36183 every 8 connection(s)
I20250114 02:52:42.802935 425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:52:42.814280 589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:52:42.816280 590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:52:42.823112 425 server_base.cc:1034] running on GCE node
W20250114 02:52:42.825536 592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:52:42.826772 425 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 02:52:42.827052 425 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 02:52:42.827261 425 hybrid_clock.cc:648] HybridClock initialized: now 1736823162827246 us; error 0 us; skew 500 ppm
I20250114 02:52:42.828043 425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:52:42.829087 585 heartbeater.cc:346] Connected to a master server at 127.0.106.126:45803
I20250114 02:52:42.829635 585 heartbeater.cc:463] Registering TS with master...
I20250114 02:52:42.830945 585 heartbeater.cc:510] Master 127.0.106.126:45803 requested a full tablet report, sending...
I20250114 02:52:42.831665 425 webserver.cc:458] Webserver started at http://127.0.106.66:38949/ using document root <none> and password file <none>
I20250114 02:52:42.832338 425 fs_manager.cc:362] Metadata directory not provided
I20250114 02:52:42.832563 425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:52:42.832863 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:52:42.834236 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-1-root/instance:
uuid: "23a08c9676c449e583b232736f562fad"
format_stamp: "Formatted at 2025-01-14 02:52:42 on dist-test-slave-k78q"
I20250114 02:52:42.834861 457 ts_manager.cc:194] Registered new tserver with Master: ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183)
I20250114 02:52:42.838495 457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50836
I20250114 02:52:42.841500 425 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250114 02:52:42.846675 597 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:52:42.847507 425 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250114 02:52:42.847752 425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-1-root
uuid: "23a08c9676c449e583b232736f562fad"
format_stamp: "Formatted at 2025-01-14 02:52:42 on dist-test-slave-k78q"
I20250114 02:52:42.848007 425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:52:42.872045 425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:52:42.873704 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:52:42.875562 425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:52:42.878588 425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:52:42.878788 425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:52:42.879045 425 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:52:42.879199 425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 02:52:42.975438 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.66:45709
I20250114 02:52:42.975517 659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.66:45709 every 8 connection(s)
I20250114 02:52:42.983356 425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:52:42.993822 663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:52:42.994691 664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:52:42.997989 660 heartbeater.cc:346] Connected to a master server at 127.0.106.126:45803
I20250114 02:52:42.998404 660 heartbeater.cc:463] Registering TS with master...
I20250114 02:52:42.999436 660 heartbeater.cc:510] Master 127.0.106.126:45803 requested a full tablet report, sending...
I20250114 02:52:43.001963 457 ts_manager.cc:194] Registered new tserver with Master: 23a08c9676c449e583b232736f562fad (127.0.106.66:45709)
I20250114 02:52:43.003198 457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50848
W20250114 02:52:43.005124 666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:52:43.006820 425 server_base.cc:1034] running on GCE node
I20250114 02:52:43.007619 425 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 02:52:43.007797 425 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 02:52:43.007937 425 hybrid_clock.cc:648] HybridClock initialized: now 1736823163007921 us; error 0 us; skew 500 ppm
I20250114 02:52:43.008425 425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:52:43.011282 425 webserver.cc:458] Webserver started at http://127.0.106.67:35587/ using document root <none> and password file <none>
I20250114 02:52:43.011703 425 fs_manager.cc:362] Metadata directory not provided
I20250114 02:52:43.011883 425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:52:43.012127 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:52:43.013159 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-2-root/instance:
uuid: "0534eb8430324a1e83c9f26c3d46887e"
format_stamp: "Formatted at 2025-01-14 02:52:43 on dist-test-slave-k78q"
I20250114 02:52:43.017495 425 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20250114 02:52:43.020704 671 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:52:43.021467 425 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 02:52:43.021713 425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-2-root
uuid: "0534eb8430324a1e83c9f26c3d46887e"
format_stamp: "Formatted at 2025-01-14 02:52:43 on dist-test-slave-k78q"
I20250114 02:52:43.022001 425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1736823160872850-425-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:52:43.042937 425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:52:43.043970 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:52:43.045867 425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:52:43.049486 425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:52:43.049682 425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:52:43.049924 425 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:52:43.050232 425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250114 02:52:43.136752 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.67:38429
I20250114 02:52:43.136813 733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.67:38429 every 8 connection(s)
I20250114 02:52:43.155870 734 heartbeater.cc:346] Connected to a master server at 127.0.106.126:45803
I20250114 02:52:43.156282 734 heartbeater.cc:463] Registering TS with master...
I20250114 02:52:43.157186 734 heartbeater.cc:510] Master 127.0.106.126:45803 requested a full tablet report, sending...
I20250114 02:52:43.159446 425 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017098986s
I20250114 02:52:43.159153 457 ts_manager.cc:194] Registered new tserver with Master: 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:43.161310 457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50854
I20250114 02:52:43.163566 425 test_util.cc:274] Using random seed: -1481796710
I20250114 02:52:43.214066 457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50858:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:52:43.217448 457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:52:43.350719 548 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8a0de188ee481e8b79fd40d711382d (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250114 02:52:43.352963 548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b8a0de188ee481e8b79fd40d711382d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.350119 550 tablet_service.cc:1467] Processing CreateTablet for tablet 287461283c6247658bdaca7184ac2067 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250114 02:52:43.357051 549 tablet_service.cc:1467] Processing CreateTablet for tablet 947325e5f22a41ed906165f55077bc67 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250114 02:52:43.358047 624 tablet_service.cc:1467] Processing CreateTablet for tablet 947325e5f22a41ed906165f55077bc67 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250114 02:52:43.357905 623 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8a0de188ee481e8b79fd40d711382d (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250114 02:52:43.364655 622 tablet_service.cc:1467] Processing CreateTablet for tablet 03c3a4647a594874889f28d2460a5566 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250114 02:52:43.374257 549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 947325e5f22a41ed906165f55077bc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.367982 699 tablet_service.cc:1467] Processing CreateTablet for tablet 287461283c6247658bdaca7184ac2067 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250114 02:52:43.368438 625 tablet_service.cc:1467] Processing CreateTablet for tablet 287461283c6247658bdaca7184ac2067 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250114 02:52:43.376657 543 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac89c2972b74708b89db61fb9f846ac (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250114 02:52:43.372385 621 tablet_service.cc:1467] Processing CreateTablet for tablet 6acde705ff2a429985e29c3cb68c983d (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250114 02:52:43.369967 619 tablet_service.cc:1467] Processing CreateTablet for tablet c2a601291dd147628db8399e31b46fe7 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250114 02:52:43.376727 546 tablet_service.cc:1467] Processing CreateTablet for tablet 6acde705ff2a429985e29c3cb68c983d (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250114 02:52:43.375015 545 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe43ac5a2984753aa6ad0f945a9d2a9 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250114 02:52:43.374401 544 tablet_service.cc:1467] Processing CreateTablet for tablet c2a601291dd147628db8399e31b46fe7 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250114 02:52:43.385708 618 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac89c2972b74708b89db61fb9f846ac (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250114 02:52:43.371747 547 tablet_service.cc:1467] Processing CreateTablet for tablet 03c3a4647a594874889f28d2460a5566 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250114 02:52:43.387518 697 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8a0de188ee481e8b79fd40d711382d (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250114 02:52:43.390470 623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b8a0de188ee481e8b79fd40d711382d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.393215 699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 287461283c6247658bdaca7184ac2067. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.381356 620 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe43ac5a2984753aa6ad0f945a9d2a9 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250114 02:52:43.381474 698 tablet_service.cc:1467] Processing CreateTablet for tablet 947325e5f22a41ed906165f55077bc67 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250114 02:52:43.390511 550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 287461283c6247658bdaca7184ac2067. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.381417 696 tablet_service.cc:1467] Processing CreateTablet for tablet 03c3a4647a594874889f28d2460a5566 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250114 02:52:43.385787 542 tablet_service.cc:1467] Processing CreateTablet for tablet eade2a70e5d54eacb0d30a272d470dfc (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250114 02:52:43.401062 625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 287461283c6247658bdaca7184ac2067. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.400336 693 tablet_service.cc:1467] Processing CreateTablet for tablet c2a601291dd147628db8399e31b46fe7 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250114 02:52:43.406487 619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a601291dd147628db8399e31b46fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.405728 697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b8a0de188ee481e8b79fd40d711382d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.407476 691 tablet_service.cc:1467] Processing CreateTablet for tablet eade2a70e5d54eacb0d30a272d470dfc (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250114 02:52:43.403753 695 tablet_service.cc:1467] Processing CreateTablet for tablet 6acde705ff2a429985e29c3cb68c983d (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250114 02:52:43.407473 617 tablet_service.cc:1467] Processing CreateTablet for tablet eade2a70e5d54eacb0d30a272d470dfc (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250114 02:52:43.402802 694 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe43ac5a2984753aa6ad0f945a9d2a9 (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250114 02:52:43.410254 545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe43ac5a2984753aa6ad0f945a9d2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.414399 543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac89c2972b74708b89db61fb9f846ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.413697 698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 947325e5f22a41ed906165f55077bc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.413798 622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03c3a4647a594874889f28d2460a5566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.416266 693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a601291dd147628db8399e31b46fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.418846 695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6acde705ff2a429985e29c3cb68c983d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.409746 692 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac89c2972b74708b89db61fb9f846ac (DEFAULT_TABLE table=test-workload [id=d5ba6509242d45ce84a1dbdfc18913b9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250114 02:52:43.422616 621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6acde705ff2a429985e29c3cb68c983d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.428567 547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03c3a4647a594874889f28d2460a5566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.439168 542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eade2a70e5d54eacb0d30a272d470dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.441596 544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a601291dd147628db8399e31b46fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.440501 696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03c3a4647a594874889f28d2460a5566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.452618 617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eade2a70e5d54eacb0d30a272d470dfc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:52:43.468178 505 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:52:43.454981 692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac89c2972b74708b89db61fb9f846ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.469852 546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6acde705ff2a429985e29c3cb68c983d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.475852 618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac89c2972b74708b89db61fb9f846ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.478047 694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe43ac5a2984753aa6ad0f945a9d2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.483218 620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe43ac5a2984753aa6ad0f945a9d2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.484962 624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 947325e5f22a41ed906165f55077bc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.493067 691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eade2a70e5d54eacb0d30a272d470dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:52:43.592646 752 tablet_bootstrap.cc:492] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62: Bootstrap starting.
I20250114 02:52:43.672175 753 tablet_bootstrap.cc:492] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad: Bootstrap starting.
I20250114 02:52:43.687446 754 tablet_bootstrap.cc:492] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e: Bootstrap starting.
I20250114 02:52:43.688373 752 tablet_bootstrap.cc:654] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:43.696380 753 tablet_bootstrap.cc:654] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:43.705858 752 tablet_bootstrap.cc:492] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62: No bootstrap required, opened a new log
I20250114 02:52:43.706308 752 ts_tablet_manager.cc:1397] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62: Time spent bootstrapping tablet: real 0.124s user 0.012s sys 0.007s
I20250114 02:52:43.706923 754 tablet_bootstrap.cc:654] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:43.709164 752 raft_consensus.cc:357] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:43.709960 752 raft_consensus.cc:383] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:43.710281 752 raft_consensus.cc:738] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Initialized, Role: FOLLOWER
I20250114 02:52:43.711647 752 consensus_queue.cc:260] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:43.715406 753 tablet_bootstrap.cc:492] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad: No bootstrap required, opened a new log
I20250114 02:52:43.715808 753 ts_tablet_manager.cc:1397] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad: Time spent bootstrapping tablet: real 0.059s user 0.013s sys 0.011s
I20250114 02:52:43.719260 585 heartbeater.cc:502] Master 127.0.106.126:45803 was elected leader, sending a full tablet report...
I20250114 02:52:43.720299 753 raft_consensus.cc:357] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:43.720995 754 tablet_bootstrap.cc:492] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e: No bootstrap required, opened a new log
I20250114 02:52:43.721010 753 raft_consensus.cc:383] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:43.721477 754 ts_tablet_manager.cc:1397] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.012s
I20250114 02:52:43.721446 753 raft_consensus.cc:738] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Initialized, Role: FOLLOWER
I20250114 02:52:43.722661 753 consensus_queue.cc:260] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:43.724635 754 raft_consensus.cc:357] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:43.725296 754 raft_consensus.cc:383] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:43.725579 754 raft_consensus.cc:738] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Initialized, Role: FOLLOWER
I20250114 02:52:43.726145 754 consensus_queue.cc:260] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:43.732496 753 ts_tablet_manager.cc:1428] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20250114 02:52:43.733199 660 heartbeater.cc:502] Master 127.0.106.126:45803 was elected leader, sending a full tablet report...
W20250114 02:52:43.733598 661 tablet.cc:2367] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 02:52:43.738452 753 tablet_bootstrap.cc:492] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad: Bootstrap starting.
I20250114 02:52:43.744565 754 ts_tablet_manager.cc:1428] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e: Time spent starting tablet: real 0.023s user 0.005s sys 0.015s
I20250114 02:52:43.745121 734 heartbeater.cc:502] Master 127.0.106.126:45803 was elected leader, sending a full tablet report...
I20250114 02:52:43.737679 752 ts_tablet_manager.cc:1428] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62: Time spent starting tablet: real 0.031s user 0.009s sys 0.000s
I20250114 02:52:43.748323 752 tablet_bootstrap.cc:492] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62: Bootstrap starting.
I20250114 02:52:43.749091 753 tablet_bootstrap.cc:654] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:43.749867 754 tablet_bootstrap.cc:492] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e: Bootstrap starting.
I20250114 02:52:43.756750 754 tablet_bootstrap.cc:654] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:43.761430 753 tablet_bootstrap.cc:492] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad: No bootstrap required, opened a new log
I20250114 02:52:43.761845 753 ts_tablet_manager.cc:1397] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.007s
I20250114 02:52:43.764449 753 raft_consensus.cc:357] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:43.765326 753 raft_consensus.cc:383] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:43.765645 753 raft_consensus.cc:738] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Initialized, Role: FOLLOWER
I20250114 02:52:43.766470 753 consensus_queue.cc:260] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:43.768949 753 ts_tablet_manager.cc:1428] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20250114 02:52:43.769127 752 tablet_bootstrap.cc:654] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:43.769848 753 tablet_bootstrap.cc:492] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad: Bootstrap starting.
I20250114 02:52:43.851321 760 raft_consensus.cc:491] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:43.851874 760 raft_consensus.cc:513] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:43.854846 760 leader_election.cc:290] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183)
I20250114 02:52:43.860983 753 tablet_bootstrap.cc:654] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:43.867259 754 tablet_bootstrap.cc:492] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e: No bootstrap required, opened a new log
I20250114 02:52:43.867386 752 tablet_bootstrap.cc:492] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62: No bootstrap required, opened a new log
I20250114 02:52:43.867977 752 ts_tablet_manager.cc:1397] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62: Time spent bootstrapping tablet: real 0.120s user 0.010s sys 0.005s
I20250114 02:52:43.869078 754 ts_tablet_manager.cc:1397] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e: Time spent bootstrapping tablet: real 0.119s user 0.026s sys 0.065s
I20250114 02:52:43.871521 754 raft_consensus.cc:357] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:43.872291 754 raft_consensus.cc:383] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:43.872601 754 raft_consensus.cc:738] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Initialized, Role: FOLLOWER
I20250114 02:52:43.873217 752 raft_consensus.cc:357] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:43.874089 752 raft_consensus.cc:383] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:43.874449 752 raft_consensus.cc:738] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Initialized, Role: FOLLOWER
I20250114 02:52:43.875240 752 consensus_queue.cc:260] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:43.873594 754 consensus_queue.cc:260] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:43.878165 754 ts_tablet_manager.cc:1428] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250114 02:52:43.881201 752 ts_tablet_manager.cc:1428] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62: Time spent starting tablet: real 0.013s user 0.006s sys 0.002s
I20250114 02:52:43.884402 752 tablet_bootstrap.cc:492] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62: Bootstrap starting.
I20250114 02:52:43.887512 754 tablet_bootstrap.cc:492] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e: Bootstrap starting.
I20250114 02:52:43.894392 754 tablet_bootstrap.cc:654] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:43.897131 752 tablet_bootstrap.cc:654] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:43.905694 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287461283c6247658bdaca7184ac2067" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62" is_pre_election: true
I20250114 02:52:43.906598 560 raft_consensus.cc:2463] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 0.
I20250114 02:52:43.908210 672 leader_election.cc:304] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:43.909006 760 raft_consensus.cc:2798] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:52:43.909349 760 raft_consensus.cc:491] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:52:43.909682 760 raft_consensus.cc:3054] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:43.911479 752 tablet_bootstrap.cc:492] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62: No bootstrap required, opened a new log
I20250114 02:52:43.913372 752 ts_tablet_manager.cc:1397] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.003s
I20250114 02:52:43.916302 760 raft_consensus.cc:513] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:43.918036 753 tablet_bootstrap.cc:492] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad: No bootstrap required, opened a new log
I20250114 02:52:43.919292 752 raft_consensus.cc:357] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:43.920084 753 ts_tablet_manager.cc:1397] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad: Time spent bootstrapping tablet: real 0.150s user 0.031s sys 0.031s
I20250114 02:52:43.919219 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287461283c6247658bdaca7184ac2067" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:43.920596 560 raft_consensus.cc:3054] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:43.921373 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287461283c6247658bdaca7184ac2067" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad" is_pre_election: true
I20250114 02:52:43.922065 635 raft_consensus.cc:2463] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 0.
I20250114 02:52:43.918285 760 leader_election.cc:290] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 election: Requested vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183)
I20250114 02:52:43.922937 753 raft_consensus.cc:357] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:43.923501 754 tablet_bootstrap.cc:492] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e: No bootstrap required, opened a new log
I20250114 02:52:43.923619 753 raft_consensus.cc:383] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:43.923929 754 ts_tablet_manager.cc:1397] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e: Time spent bootstrapping tablet: real 0.037s user 0.020s sys 0.003s
I20250114 02:52:43.924024 753 raft_consensus.cc:738] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Initialized, Role: FOLLOWER
I20250114 02:52:43.925107 753 consensus_queue.cc:260] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:43.927346 754 raft_consensus.cc:357] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:43.927816 753 ts_tablet_manager.cc:1428] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250114 02:52:43.928049 754 raft_consensus.cc:383] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:43.928047 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287461283c6247658bdaca7184ac2067" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:52:43.928512 754 raft_consensus.cc:738] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Initialized, Role: FOLLOWER
I20250114 02:52:43.928771 753 tablet_bootstrap.cc:492] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad: Bootstrap starting.
I20250114 02:52:43.929195 754 consensus_queue.cc:260] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:43.928726 634 raft_consensus.cc:3054] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:43.931298 754 ts_tablet_manager.cc:1428] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20250114 02:52:43.932116 754 tablet_bootstrap.cc:492] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e: Bootstrap starting.
I20250114 02:52:43.920030 752 raft_consensus.cc:383] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:43.937031 634 raft_consensus.cc:2463] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 1.
I20250114 02:52:43.938135 752 raft_consensus.cc:738] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Initialized, Role: FOLLOWER
I20250114 02:52:43.938841 674 leader_election.cc:304] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad; no voters:
I20250114 02:52:43.938832 752 consensus_queue.cc:260] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:43.939721 760 raft_consensus.cc:2798] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:52:43.941423 560 raft_consensus.cc:2463] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 1.
I20250114 02:52:43.943394 752 ts_tablet_manager.cc:1428] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62: Time spent starting tablet: real 0.030s user 0.003s sys 0.003s
I20250114 02:52:43.944172 752 tablet_bootstrap.cc:492] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62: Bootstrap starting.
I20250114 02:52:43.949782 759 raft_consensus.cc:491] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:43.950315 759 raft_consensus.cc:513] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:43.954197 759 leader_election.cc:290] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:43.957913 760 raft_consensus.cc:695] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 1 LEADER]: Becoming Leader. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Running, Role: LEADER
I20250114 02:52:43.963351 753 tablet_bootstrap.cc:654] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:43.965953 760 consensus_queue.cc:237] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:43.975426 754 tablet_bootstrap.cc:654] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:43.981303 752 tablet_bootstrap.cc:654] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:43.998951 753 tablet_bootstrap.cc:492] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad: No bootstrap required, opened a new log
I20250114 02:52:43.998925 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe43ac5a2984753aa6ad0f945a9d2a9" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62" is_pre_election: true
I20250114 02:52:43.999795 753 ts_tablet_manager.cc:1397] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad: Time spent bootstrapping tablet: real 0.071s user 0.015s sys 0.002s
W20250114 02:52:44.000998 598 leader_election.cc:343] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:52:44.003077 753 raft_consensus.cc:357] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.006048 753 raft_consensus.cc:383] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.006237 754 tablet_bootstrap.cc:492] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e: No bootstrap required, opened a new log
I20250114 02:52:44.006654 754 ts_tablet_manager.cc:1397] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e: Time spent bootstrapping tablet: real 0.075s user 0.009s sys 0.006s
I20250114 02:52:44.006450 753 raft_consensus.cc:738] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.007694 753 consensus_queue.cc:260] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.008422 752 tablet_bootstrap.cc:492] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62: No bootstrap required, opened a new log
I20250114 02:52:44.008842 752 ts_tablet_manager.cc:1397] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62: Time spent bootstrapping tablet: real 0.065s user 0.015s sys 0.005s
I20250114 02:52:44.010155 753 ts_tablet_manager.cc:1428] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250114 02:52:44.012758 753 tablet_bootstrap.cc:492] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad: Bootstrap starting.
I20250114 02:52:44.014552 754 raft_consensus.cc:357] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.015131 754 raft_consensus.cc:383] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.014957 752 raft_consensus.cc:357] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.015416 754 raft_consensus.cc:738] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.015599 752 raft_consensus.cc:383] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.015908 752 raft_consensus.cc:738] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.016566 754 consensus_queue.cc:260] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.016882 752 consensus_queue.cc:260] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.020608 752 ts_tablet_manager.cc:1428] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20250114 02:52:44.017860 454 catalog_manager.cc:5526] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67). New cstate: current_term: 1 leader_uuid: "0534eb8430324a1e83c9f26c3d46887e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:44.025784 760 raft_consensus.cc:491] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.026305 760 raft_consensus.cc:513] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.025749 752 tablet_bootstrap.cc:492] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62: Bootstrap starting.
I20250114 02:52:44.031342 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947325e5f22a41ed906165f55077bc67" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62" is_pre_election: true
I20250114 02:52:44.031925 560 raft_consensus.cc:2463] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 0.
I20250114 02:52:44.032989 672 leader_election.cc:304] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:44.033254 753 tablet_bootstrap.cc:654] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.033910 771 raft_consensus.cc:2798] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:52:44.034276 771 raft_consensus.cc:491] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:52:44.034602 771 raft_consensus.cc:3054] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.035161 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947325e5f22a41ed906165f55077bc67" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad" is_pre_election: true
I20250114 02:52:44.033936 760 leader_election.cc:290] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 23a08c9676c449e583b232736f562fad (127.0.106.66:45709)
W20250114 02:52:44.037576 674 leader_election.cc:343] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23a08c9676c449e583b232736f562fad (127.0.106.66:45709): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:52:44.039932 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe43ac5a2984753aa6ad0f945a9d2a9" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e" is_pre_election: true
I20250114 02:52:44.040572 754 ts_tablet_manager.cc:1428] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e: Time spent starting tablet: real 0.034s user 0.006s sys 0.002s
I20250114 02:52:44.040661 709 raft_consensus.cc:2463] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 0.
I20250114 02:52:44.041584 754 tablet_bootstrap.cc:492] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e: Bootstrap starting.
I20250114 02:52:44.042119 601 leader_election.cc:304] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad; no voters: ef76faab133b4cbf96d9e7610be21b62
I20250114 02:52:44.043457 759 raft_consensus.cc:2798] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:52:44.043788 759 raft_consensus.cc:491] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:52:44.044093 759 raft_consensus.cc:3054] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.049057 760 raft_consensus.cc:491] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.049477 760 raft_consensus.cc:513] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.048542 771 raft_consensus.cc:513] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.050493 759 raft_consensus.cc:513] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.051465 771 leader_election.cc:290] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 23a08c9676c449e583b232736f562fad (127.0.106.66:45709)
I20250114 02:52:44.051985 760 leader_election.cc:290] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 23a08c9676c449e583b232736f562fad (127.0.106.66:45709)
I20250114 02:52:44.054864 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947325e5f22a41ed906165f55077bc67" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:52:44.055235 752 tablet_bootstrap.cc:654] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.054379 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acde705ff2a429985e29c3cb68c983d" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad" is_pre_election: true
I20250114 02:52:44.052610 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acde705ff2a429985e29c3cb68c983d" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62" is_pre_election: true
I20250114 02:52:44.052743 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947325e5f22a41ed906165f55077bc67" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:44.057062 559 raft_consensus.cc:3054] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:52:44.061674 672 leader_election.cc:343] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:52:44.062299 674 leader_election.cc:343] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23a08c9676c449e583b232736f562fad (127.0.106.66:45709): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:52:44.062829 674 leader_election.cc:343] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23a08c9676c449e583b232736f562fad (127.0.106.66:45709): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:52:44.063133 674 leader_election.cc:304] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e; no voters: 23a08c9676c449e583b232736f562fad, ef76faab133b4cbf96d9e7610be21b62
I20250114 02:52:44.063952 760 raft_consensus.cc:2743] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:52:44.065557 758 raft_consensus.cc:491] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.066100 758 raft_consensus.cc:513] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.067023 559 raft_consensus.cc:2463] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 1.
I20250114 02:52:44.068179 672 leader_election.cc:304] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters: 23a08c9676c449e583b232736f562fad
I20250114 02:52:44.068938 760 raft_consensus.cc:2798] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:52:44.069393 760 raft_consensus.cc:695] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 1 LEADER]: Becoming Leader. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Running, Role: LEADER
I20250114 02:52:44.070709 760 consensus_queue.cc:237] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.071334 758 leader_election.cc:290] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.074177 759 leader_election.cc:290] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.075309 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe43ac5a2984753aa6ad0f945a9d2a9" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:44.081211 709 raft_consensus.cc:3054] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.080729 754 tablet_bootstrap.cc:654] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.086035 759 raft_consensus.cc:491] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.086506 759 raft_consensus.cc:513] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.092968 759 leader_election.cc:290] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.094806 708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e" is_pre_election: true
W20250114 02:52:44.097188 601 leader_election.cc:343] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:52:44.098655 709 raft_consensus.cc:2463] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 1.
I20250114 02:52:44.100528 601 leader_election.cc:304] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad; no voters:
I20250114 02:52:44.105015 753 tablet_bootstrap.cc:492] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad: No bootstrap required, opened a new log
I20250114 02:52:44.105690 753 ts_tablet_manager.cc:1397] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad: Time spent bootstrapping tablet: real 0.093s user 0.009s sys 0.011s
I20250114 02:52:44.114167 759 raft_consensus.cc:2798] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:52:44.120625 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe43ac5a2984753aa6ad0f945a9d2a9" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:44.128602 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62" is_pre_election: true
W20250114 02:52:44.129148 598 leader_election.cc:343] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:52:44.135485 559 raft_consensus.cc:2463] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 0.
I20250114 02:52:44.138010 598 leader_election.cc:304] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23a08c9676c449e583b232736f562fad, ef76faab133b4cbf96d9e7610be21b62; no voters: 0534eb8430324a1e83c9f26c3d46887e
I20250114 02:52:44.141872 759 raft_consensus.cc:695] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Becoming Leader. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:44.143517 752 tablet_bootstrap.cc:492] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62: No bootstrap required, opened a new log
I20250114 02:52:44.144037 752 ts_tablet_manager.cc:1397] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62: Time spent bootstrapping tablet: real 0.122s user 0.017s sys 0.002s
I20250114 02:52:44.143573 759 consensus_queue.cc:237] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.149663 787 raft_consensus.cc:2798] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:52:44.149835 754 tablet_bootstrap.cc:492] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e: No bootstrap required, opened a new log
I20250114 02:52:44.150794 752 raft_consensus.cc:357] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.147917 753 raft_consensus.cc:357] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.152717 752 raft_consensus.cc:383] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.151722 754 ts_tablet_manager.cc:1397] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e: Time spent bootstrapping tablet: real 0.110s user 0.014s sys 0.008s
I20250114 02:52:44.154150 753 raft_consensus.cc:383] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.154597 752 raft_consensus.cc:738] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.154842 787 raft_consensus.cc:491] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:52:44.158739 754 raft_consensus.cc:357] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.160257 787 raft_consensus.cc:3054] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.161527 754 raft_consensus.cc:383] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.160566 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad" is_pre_election: true
I20250114 02:52:44.161875 754 raft_consensus.cc:738] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.159713 752 consensus_queue.cc:260] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.158080 753 raft_consensus.cc:738] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.165323 752 ts_tablet_manager.cc:1428] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62: Time spent starting tablet: real 0.021s user 0.004s sys 0.003s
I20250114 02:52:44.166285 752 tablet_bootstrap.cc:492] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62: Bootstrap starting.
I20250114 02:52:44.162596 754 consensus_queue.cc:260] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.171363 753 consensus_queue.cc:260] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.172618 754 ts_tablet_manager.cc:1428] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e: Time spent starting tablet: real 0.018s user 0.008s sys 0.001s
I20250114 02:52:44.173444 754 tablet_bootstrap.cc:492] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e: Bootstrap starting.
I20250114 02:52:44.174351 753 ts_tablet_manager.cc:1428] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad: Time spent starting tablet: real 0.068s user 0.005s sys 0.005s
I20250114 02:52:44.175367 753 tablet_bootstrap.cc:492] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad: Bootstrap starting.
I20250114 02:52:44.176297 457 catalog_manager.cc:5526] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad reported cstate change: term changed from 0 to 1, leader changed from <none> to 23a08c9676c449e583b232736f562fad (127.0.106.66). New cstate: current_term: 1 leader_uuid: "23a08c9676c449e583b232736f562fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:44.180523 754 tablet_bootstrap.cc:654] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.182865 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e" is_pre_election: true
I20250114 02:52:44.183485 709 raft_consensus.cc:2463] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 0.
I20250114 02:52:44.186298 526 leader_election.cc:304] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:44.187098 758 raft_consensus.cc:2798] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:52:44.187572 758 raft_consensus.cc:491] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:52:44.188223 758 raft_consensus.cc:3054] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.181447 787 raft_consensus.cc:513] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.192131 635 raft_consensus.cc:2388] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef76faab133b4cbf96d9e7610be21b62 in current term 1: Already voted for candidate 23a08c9676c449e583b232736f562fad in this term.
I20250114 02:52:44.204046 787 leader_election.cc:290] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.204389 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:44.205816 753 tablet_bootstrap.cc:654] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.203358 758 raft_consensus.cc:513] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.212878 559 raft_consensus.cc:2388] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 23a08c9676c449e583b232736f562fad in current term 1: Already voted for candidate ef76faab133b4cbf96d9e7610be21b62 in this term.
I20250114 02:52:44.231066 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:44.232184 709 raft_consensus.cc:3054] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.232828 758 leader_election.cc:290] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 election: Requested vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.236120 708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:44.234972 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:52:44.240425 635 raft_consensus.cc:2388] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ef76faab133b4cbf96d9e7610be21b62 in current term 1: Already voted for candidate 23a08c9676c449e583b232736f562fad in this term.
I20250114 02:52:44.241274 454 catalog_manager.cc:5526] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67). New cstate: current_term: 1 leader_uuid: "0534eb8430324a1e83c9f26c3d46887e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: HEALTHY } } }
I20250114 02:52:44.243875 525 leader_election.cc:304] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef76faab133b4cbf96d9e7610be21b62; no voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad
I20250114 02:52:44.245672 758 raft_consensus.cc:2743] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 02:52:44.246768 752 tablet_bootstrap.cc:654] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.250108 754 tablet_bootstrap.cc:492] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e: No bootstrap required, opened a new log
I20250114 02:52:44.251634 754 ts_tablet_manager.cc:1397] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.003s
I20250114 02:52:44.255033 753 tablet_bootstrap.cc:492] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad: No bootstrap required, opened a new log
I20250114 02:52:44.255460 753 ts_tablet_manager.cc:1397] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad: Time spent bootstrapping tablet: real 0.080s user 0.007s sys 0.009s
I20250114 02:52:44.257407 709 raft_consensus.cc:2463] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 1.
I20250114 02:52:44.258337 601 leader_election.cc:304] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad; no voters: ef76faab133b4cbf96d9e7610be21b62
I20250114 02:52:44.255525 754 raft_consensus.cc:357] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.259217 754 raft_consensus.cc:383] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.259516 754 raft_consensus.cc:738] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.259630 787 raft_consensus.cc:2798] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:52:44.260241 787 raft_consensus.cc:695] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Becoming Leader. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:44.259750 753 raft_consensus.cc:357] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.260571 754 consensus_queue.cc:260] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.261039 758 raft_consensus.cc:491] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.261245 753 raft_consensus.cc:383] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.260934 787 consensus_queue.cc:237] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.261442 758 raft_consensus.cc:513] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.263093 758 leader_election.cc:290] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.262992 754 ts_tablet_manager.cc:1428] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e: Time spent starting tablet: real 0.011s user 0.003s sys 0.004s
I20250114 02:52:44.264307 754 tablet_bootstrap.cc:492] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e: Bootstrap starting.
I20250114 02:52:44.264859 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad" is_pre_election: true
I20250114 02:52:44.264956 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e" is_pre_election: true
I20250114 02:52:44.260916 752 tablet_bootstrap.cc:492] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62: No bootstrap required, opened a new log
I20250114 02:52:44.266014 752 ts_tablet_manager.cc:1397] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62: Time spent bootstrapping tablet: real 0.100s user 0.004s sys 0.010s
I20250114 02:52:44.265944 709 raft_consensus.cc:2463] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 0.
W20250114 02:52:44.266342 525 leader_election.cc:343] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23a08c9676c449e583b232736f562fad (127.0.106.66:45709): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:52:44.266856 526 leader_election.cc:304] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters: 23a08c9676c449e583b232736f562fad
I20250114 02:52:44.268042 758 raft_consensus.cc:2798] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:52:44.268402 758 raft_consensus.cc:491] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:52:44.268702 758 raft_consensus.cc:3054] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.269418 752 raft_consensus.cc:357] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.270098 752 raft_consensus.cc:383] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.270318 752 raft_consensus.cc:738] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.272614 754 tablet_bootstrap.cc:654] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.261572 753 raft_consensus.cc:738] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.273738 753 consensus_queue.cc:260] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.276273 753 ts_tablet_manager.cc:1428] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20250114 02:52:44.277375 753 tablet_bootstrap.cc:492] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad: Bootstrap starting.
I20250114 02:52:44.279032 752 consensus_queue.cc:260] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.280136 758 raft_consensus.cc:513] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.282748 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad"
W20250114 02:52:44.283910 525 leader_election.cc:343] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23a08c9676c449e583b232736f562fad (127.0.106.66:45709): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:52:44.284776 758 leader_election.cc:290] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 election: Requested vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.287457 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:44.288025 754 tablet_bootstrap.cc:492] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e: No bootstrap required, opened a new log
I20250114 02:52:44.288429 754 ts_tablet_manager.cc:1397] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.016s
I20250114 02:52:44.287937 709 raft_consensus.cc:3054] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.289259 457 catalog_manager.cc:5526] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad reported cstate change: term changed from 0 to 1, leader changed from <none> to 23a08c9676c449e583b232736f562fad (127.0.106.66). New cstate: current_term: 1 leader_uuid: "23a08c9676c449e583b232736f562fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:44.291210 754 raft_consensus.cc:357] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.291862 754 raft_consensus.cc:383] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.292110 754 raft_consensus.cc:738] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.295931 752 ts_tablet_manager.cc:1428] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62: Time spent starting tablet: real 0.030s user 0.003s sys 0.004s
I20250114 02:52:44.296578 752 tablet_bootstrap.cc:492] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62: Bootstrap starting.
I20250114 02:52:44.296551 709 raft_consensus.cc:2463] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 1.
I20250114 02:52:44.297677 526 leader_election.cc:304] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters: 23a08c9676c449e583b232736f562fad
I20250114 02:52:44.298137 753 tablet_bootstrap.cc:654] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.295111 754 consensus_queue.cc:260] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.300585 797 raft_consensus.cc:2798] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:52:44.302841 797 raft_consensus.cc:695] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 1 LEADER]: Becoming Leader. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:52:44.303383 752 tablet_bootstrap.cc:654] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.306319 754 ts_tablet_manager.cc:1428] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e: Time spent starting tablet: real 0.018s user 0.005s sys 0.002s
I20250114 02:52:44.309540 754 tablet_bootstrap.cc:492] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e: Bootstrap starting.
I20250114 02:52:44.312752 758 raft_consensus.cc:491] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.313117 758 raft_consensus.cc:513] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.315950 758 leader_election.cc:290] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.316649 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad" is_pre_election: true
I20250114 02:52:44.304379 797 consensus_queue.cc:237] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.317217 635 raft_consensus.cc:2463] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 0.
I20250114 02:52:44.319474 525 leader_election.cc:304] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a08c9676c449e583b232736f562fad, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:44.317224 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e" is_pre_election: true
I20250114 02:52:44.320353 758 raft_consensus.cc:2798] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250114 02:52:44.323241 526 leader_election.cc:343] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:52:44.323236 760 raft_consensus.cc:491] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.322903 758 raft_consensus.cc:491] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:52:44.323881 760 raft_consensus.cc:513] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.324365 758 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.325898 760 leader_election.cc:290] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 23a08c9676c449e583b232736f562fad (127.0.106.66:45709)
I20250114 02:52:44.326406 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade2a70e5d54eacb0d30a272d470dfc" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62" is_pre_election: true
I20250114 02:52:44.326738 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade2a70e5d54eacb0d30a272d470dfc" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad" is_pre_election: true
W20250114 02:52:44.327593 672 leader_election.cc:343] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:52:44.327992 674 leader_election.cc:343] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23a08c9676c449e583b232736f562fad (127.0.106.66:45709): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:52:44.328408 674 leader_election.cc:304] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e; no voters: 23a08c9676c449e583b232736f562fad, ef76faab133b4cbf96d9e7610be21b62
I20250114 02:52:44.329116 760 raft_consensus.cc:2743] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:52:44.331593 754 tablet_bootstrap.cc:654] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.334692 753 tablet_bootstrap.cc:492] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad: No bootstrap required, opened a new log
I20250114 02:52:44.334960 753 ts_tablet_manager.cc:1397] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad: Time spent bootstrapping tablet: real 0.058s user 0.014s sys 0.008s
I20250114 02:52:44.336277 758 raft_consensus.cc:513] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.340366 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:44.340420 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:52:44.341091 635 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:52:44.341584 526 leader_election.cc:343] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:52:44.342855 758 leader_election.cc:290] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 election: Requested vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.346249 752 tablet_bootstrap.cc:492] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62: No bootstrap required, opened a new log
I20250114 02:52:44.348824 754 tablet_bootstrap.cc:492] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e: No bootstrap required, opened a new log
I20250114 02:52:44.349228 754 ts_tablet_manager.cc:1397] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.012s
I20250114 02:52:44.351056 752 ts_tablet_manager.cc:1397] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62: Time spent bootstrapping tablet: real 0.055s user 0.017s sys 0.001s
I20250114 02:52:44.352527 635 raft_consensus.cc:2463] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 1.
I20250114 02:52:44.353224 525 leader_election.cc:304] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23a08c9676c449e583b232736f562fad, ef76faab133b4cbf96d9e7610be21b62; no voters: 0534eb8430324a1e83c9f26c3d46887e
I20250114 02:52:44.353786 797 raft_consensus.cc:2798] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:52:44.353157 454 catalog_manager.cc:5526] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65). New cstate: current_term: 1 leader_uuid: "ef76faab133b4cbf96d9e7610be21b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:44.353675 752 raft_consensus.cc:357] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.354185 797 raft_consensus.cc:695] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 1 LEADER]: Becoming Leader. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:52:44.354254 752 raft_consensus.cc:383] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.354621 752 raft_consensus.cc:738] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.355023 797 consensus_queue.cc:237] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.355659 752 consensus_queue.cc:260] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.357187 754 raft_consensus.cc:357] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.357779 754 raft_consensus.cc:383] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.351605 753 raft_consensus.cc:357] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.357982 754 raft_consensus.cc:738] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.358251 753 raft_consensus.cc:383] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.358601 754 consensus_queue.cc:260] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.358552 753 raft_consensus.cc:738] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.361068 753 consensus_queue.cc:260] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.361521 754 ts_tablet_manager.cc:1428] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20250114 02:52:44.366263 754 tablet_bootstrap.cc:492] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e: Bootstrap starting.
I20250114 02:52:44.374679 753 ts_tablet_manager.cc:1428] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad: Time spent starting tablet: real 0.039s user 0.006s sys 0.001s
I20250114 02:52:44.376224 753 tablet_bootstrap.cc:492] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad: Bootstrap starting.
I20250114 02:52:44.378265 752 ts_tablet_manager.cc:1428] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62: Time spent starting tablet: real 0.027s user 0.010s sys 0.004s
I20250114 02:52:44.379135 752 tablet_bootstrap.cc:492] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62: Bootstrap starting.
I20250114 02:52:44.383668 754 tablet_bootstrap.cc:654] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.385397 752 tablet_bootstrap.cc:654] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.387488 753 tablet_bootstrap.cc:654] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.387275 456 catalog_manager.cc:5526] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65). New cstate: current_term: 1 leader_uuid: "ef76faab133b4cbf96d9e7610be21b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:44.393090 753 tablet_bootstrap.cc:492] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad: No bootstrap required, opened a new log
I20250114 02:52:44.393476 753 ts_tablet_manager.cc:1397] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.001s
I20250114 02:52:44.394114 754 tablet_bootstrap.cc:492] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e: No bootstrap required, opened a new log
I20250114 02:52:44.394462 754 ts_tablet_manager.cc:1397] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20250114 02:52:44.395820 753 raft_consensus.cc:357] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.396451 753 raft_consensus.cc:383] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.396672 753 raft_consensus.cc:738] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.397169 753 consensus_queue.cc:260] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.397411 754 raft_consensus.cc:357] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.398005 754 raft_consensus.cc:383] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.398238 754 raft_consensus.cc:738] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.399158 753 ts_tablet_manager.cc:1428] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20250114 02:52:44.398931 754 consensus_queue.cc:260] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.399904 753 tablet_bootstrap.cc:492] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad: Bootstrap starting.
I20250114 02:52:44.400430 754 ts_tablet_manager.cc:1428] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:52:44.406085 753 tablet_bootstrap.cc:654] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.414676 752 tablet_bootstrap.cc:492] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62: No bootstrap required, opened a new log
I20250114 02:52:44.415056 752 ts_tablet_manager.cc:1397] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.012s
I20250114 02:52:44.418485 760 consensus_queue.cc:1035] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:44.419853 752 raft_consensus.cc:357] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.420536 752 raft_consensus.cc:383] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.420796 752 raft_consensus.cc:738] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.421411 752 consensus_queue.cc:260] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.423892 752 ts_tablet_manager.cc:1428] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20250114 02:52:44.424831 752 tablet_bootstrap.cc:492] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62: Bootstrap starting.
I20250114 02:52:44.433257 752 tablet_bootstrap.cc:654] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62: Neither blocks nor log segments found. Creating new log.
I20250114 02:52:44.437552 753 tablet_bootstrap.cc:492] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad: No bootstrap required, opened a new log
I20250114 02:52:44.437929 753 ts_tablet_manager.cc:1397] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.023s
I20250114 02:52:44.447060 760 consensus_queue.cc:1035] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:44.448292 753 raft_consensus.cc:357] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.449198 753 raft_consensus.cc:383] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.449544 753 raft_consensus.cc:738] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.450102 752 tablet_bootstrap.cc:492] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62: No bootstrap required, opened a new log
I20250114 02:52:44.450500 752 ts_tablet_manager.cc:1397] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.009s
I20250114 02:52:44.450201 753 consensus_queue.cc:260] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.456985 752 raft_consensus.cc:357] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.457895 752 raft_consensus.cc:383] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:52:44.459372 752 raft_consensus.cc:738] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Initialized, Role: FOLLOWER
I20250114 02:52:44.460090 752 consensus_queue.cc:260] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.459113 753 ts_tablet_manager.cc:1428] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad: Time spent starting tablet: real 0.021s user 0.005s sys 0.004s
I20250114 02:52:44.464195 752 ts_tablet_manager.cc:1428] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250114 02:52:44.461992 787 raft_consensus.cc:491] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.465029 787 raft_consensus.cc:513] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.466450 787 leader_election.cc:290] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.467586 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947325e5f22a41ed906165f55077bc67" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e" is_pre_election: true
I20250114 02:52:44.468779 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947325e5f22a41ed906165f55077bc67" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62" is_pre_election: true
I20250114 02:52:44.469558 559 raft_consensus.cc:2388] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23a08c9676c449e583b232736f562fad in current term 1: Already voted for candidate 0534eb8430324a1e83c9f26c3d46887e in this term.
I20250114 02:52:44.470726 598 leader_election.cc:304] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23a08c9676c449e583b232736f562fad; no voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62
I20250114 02:52:44.471675 787 raft_consensus.cc:3054] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.474722 759 raft_consensus.cc:491] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.475100 759 raft_consensus.cc:513] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.476733 759 leader_election.cc:290] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
W20250114 02:52:44.477396 505 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:52:44.477566 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c3a4647a594874889f28d2460a5566" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e" is_pre_election: true
I20250114 02:52:44.478065 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c3a4647a594874889f28d2460a5566" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62" is_pre_election: true
I20250114 02:52:44.478119 709 raft_consensus.cc:2463] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 0.
I20250114 02:52:44.479151 559 raft_consensus.cc:2463] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 0.
I20250114 02:52:44.479359 601 leader_election.cc:304] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad; no voters:
I20250114 02:52:44.480942 759 raft_consensus.cc:2798] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:52:44.481217 759 raft_consensus.cc:491] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:52:44.481448 759 raft_consensus.cc:3054] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.479894 787 raft_consensus.cc:2743] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:52:44.485107 759 raft_consensus.cc:513] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.488193 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c3a4647a594874889f28d2460a5566" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:44.488471 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c3a4647a594874889f28d2460a5566" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:44.489180 709 raft_consensus.cc:3054] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.488909 559 raft_consensus.cc:3054] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.494488 709 raft_consensus.cc:2463] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 1.
I20250114 02:52:44.495819 601 leader_election.cc:304] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad; no voters:
I20250114 02:52:44.496547 787 raft_consensus.cc:2798] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:52:44.496965 787 raft_consensus.cc:695] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Becoming Leader. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:44.497016 759 leader_election.cc:290] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.497654 559 raft_consensus.cc:2463] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 1.
I20250114 02:52:44.497700 787 consensus_queue.cc:237] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.504489 455 catalog_manager.cc:5526] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad reported cstate change: term changed from 0 to 1, leader changed from <none> to 23a08c9676c449e583b232736f562fad (127.0.106.66). New cstate: current_term: 1 leader_uuid: "23a08c9676c449e583b232736f562fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:44.572192 760 consensus_queue.cc:1035] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:44.582468 771 consensus_queue.cc:1035] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:44.605878 787 raft_consensus.cc:491] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.606215 787 raft_consensus.cc:513] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.607553 787 leader_election.cc:290] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.608222 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade2a70e5d54eacb0d30a272d470dfc" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62" is_pre_election: true
I20250114 02:52:44.608783 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade2a70e5d54eacb0d30a272d470dfc" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e" is_pre_election: true
I20250114 02:52:44.608875 559 raft_consensus.cc:2463] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 0.
I20250114 02:52:44.609287 709 raft_consensus.cc:2463] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 0.
I20250114 02:52:44.609791 598 leader_election.cc:304] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a08c9676c449e583b232736f562fad, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:44.610654 787 raft_consensus.cc:2798] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:52:44.610916 787 raft_consensus.cc:491] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:52:44.611161 787 raft_consensus.cc:3054] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.614753 787 raft_consensus.cc:513] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.615954 787 leader_election.cc:290] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.616686 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade2a70e5d54eacb0d30a272d470dfc" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:44.616816 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade2a70e5d54eacb0d30a272d470dfc" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:44.617264 709 raft_consensus.cc:3054] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.617228 559 raft_consensus.cc:3054] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.621502 709 raft_consensus.cc:2463] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 1.
I20250114 02:52:44.622321 601 leader_election.cc:304] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad; no voters:
I20250114 02:52:44.623313 559 raft_consensus.cc:2463] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 1.
I20250114 02:52:44.623133 759 raft_consensus.cc:2798] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:52:44.624115 759 raft_consensus.cc:695] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Becoming Leader. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:44.622709 787 raft_consensus.cc:491] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.624900 759 consensus_queue.cc:237] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.624899 787 raft_consensus.cc:513] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.626816 787 leader_election.cc:290] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.627597 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62" is_pre_election: true
I20250114 02:52:44.629217 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e" is_pre_election: true
I20250114 02:52:44.629973 709 raft_consensus.cc:2388] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23a08c9676c449e583b232736f562fad in current term 1: Already voted for candidate ef76faab133b4cbf96d9e7610be21b62 in this term.
I20250114 02:52:44.631013 601 leader_election.cc:304] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23a08c9676c449e583b232736f562fad; no voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62
I20250114 02:52:44.632088 759 raft_consensus.cc:3054] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.636366 759 raft_consensus.cc:2743] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:52:44.635782 456 catalog_manager.cc:5526] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad reported cstate change: term changed from 0 to 1, leader changed from <none> to 23a08c9676c449e583b232736f562fad (127.0.106.66). New cstate: current_term: 1 leader_uuid: "23a08c9676c449e583b232736f562fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:44.652629 803 raft_consensus.cc:491] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.653080 803 raft_consensus.cc:513] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.654647 803 leader_election.cc:290] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.655403 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe43ac5a2984753aa6ad0f945a9d2a9" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad" is_pre_election: true
I20250114 02:52:44.655616 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe43ac5a2984753aa6ad0f945a9d2a9" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e" is_pre_election: true
I20250114 02:52:44.656411 709 raft_consensus.cc:2388] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef76faab133b4cbf96d9e7610be21b62 in current term 1: Already voted for candidate 23a08c9676c449e583b232736f562fad in this term.
I20250114 02:52:44.657230 526 leader_election.cc:304] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef76faab133b4cbf96d9e7610be21b62; no voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad
I20250114 02:52:44.657892 803 raft_consensus.cc:3054] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.662317 803 raft_consensus.cc:2743] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:52:44.690701 759 consensus_queue.cc:1035] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:44.700906 759 consensus_queue.cc:1035] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:44.707679 771 raft_consensus.cc:491] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.708158 771 raft_consensus.cc:513] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.710098 771 leader_election.cc:290] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 23a08c9676c449e583b232736f562fad (127.0.106.66:45709)
I20250114 02:52:44.711244 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62" is_pre_election: true
I20250114 02:52:44.711563 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a08c9676c449e583b232736f562fad" is_pre_election: true
I20250114 02:52:44.712268 635 raft_consensus.cc:2388] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0534eb8430324a1e83c9f26c3d46887e in current term 1: Already voted for candidate ef76faab133b4cbf96d9e7610be21b62 in this term.
I20250114 02:52:44.713268 674 leader_election.cc:304] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e; no voters: 23a08c9676c449e583b232736f562fad, ef76faab133b4cbf96d9e7610be21b62
I20250114 02:52:44.713990 771 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.715075 786 consensus_queue.cc:1035] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:52:44.719369 771 raft_consensus.cc:2743] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:52:44.724103 786 consensus_queue.cc:1035] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:44.732270 786 raft_consensus.cc:491] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:52:44.732679 786 raft_consensus.cc:513] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.734411 786 leader_election.cc:290] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.736649 708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acde705ff2a429985e29c3cb68c983d" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e" is_pre_election: true
I20250114 02:52:44.737213 708 raft_consensus.cc:2463] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 0.
I20250114 02:52:44.739073 601 leader_election.cc:304] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad; no voters:
I20250114 02:52:44.739881 787 raft_consensus.cc:2798] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:52:44.740255 787 raft_consensus.cc:491] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:52:44.740533 787 raft_consensus.cc:3054] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.745299 803 consensus_queue.cc:1035] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:44.750643 803 consensus_queue.cc:1035] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:44.750739 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acde705ff2a429985e29c3cb68c983d" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62" is_pre_election: true
I20250114 02:52:44.751915 559 raft_consensus.cc:2463] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 0.
I20250114 02:52:44.754669 787 raft_consensus.cc:513] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.757639 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acde705ff2a429985e29c3cb68c983d" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:44.757937 707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acde705ff2a429985e29c3cb68c983d" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:44.758194 559 raft_consensus.cc:3054] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.758431 707 raft_consensus.cc:3054] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:52:44.763151 787 leader_election.cc:290] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:44.764299 707 raft_consensus.cc:2463] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 1.
I20250114 02:52:44.765125 559 raft_consensus.cc:2463] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 1.
I20250114 02:52:44.765206 601 leader_election.cc:304] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad; no voters:
I20250114 02:52:44.766100 787 raft_consensus.cc:2798] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:52:44.766769 787 raft_consensus.cc:695] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Becoming Leader. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:44.767557 787 consensus_queue.cc:237] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:44.792892 457 catalog_manager.cc:5526] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad reported cstate change: term changed from 0 to 1, leader changed from <none> to 23a08c9676c449e583b232736f562fad (127.0.106.66). New cstate: current_term: 1 leader_uuid: "23a08c9676c449e583b232736f562fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:44.801419 758 consensus_queue.cc:1035] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:44.807771 758 consensus_queue.cc:1035] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:44.938591 787 consensus_queue.cc:1035] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:44.947986 786 consensus_queue.cc:1035] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:52:45.063252 786 consensus_queue.cc:1035] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:45.075413 759 consensus_queue.cc:1035] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:45.143666 709 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "287461283c6247658bdaca7184ac2067"
dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
mode: GRACEFUL
new_leader_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:38728
I20250114 02:52:45.144399 709 raft_consensus.cc:604] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 1 LEADER]: Received request to transfer leadership to TS ef76faab133b4cbf96d9e7610be21b62
I20250114 02:52:45.149019 709 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "947325e5f22a41ed906165f55077bc67"
dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
mode: GRACEFUL
new_leader_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:38728
I20250114 02:52:45.149639 709 raft_consensus.cc:604] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 1 LEADER]: Received request to transfer leadership to TS ef76faab133b4cbf96d9e7610be21b62
I20250114 02:52:45.170030 635 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d"
dest_uuid: "23a08c9676c449e583b232736f562fad"
mode: GRACEFUL
new_leader_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:50686
I20250114 02:52:45.170446 635 raft_consensus.cc:604] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Received request to transfer leadership to TS ef76faab133b4cbf96d9e7610be21b62
I20250114 02:52:45.174360 635 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03c3a4647a594874889f28d2460a5566"
dest_uuid: "23a08c9676c449e583b232736f562fad"
mode: GRACEFUL
new_leader_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:50686
I20250114 02:52:45.174779 635 raft_consensus.cc:604] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Received request to transfer leadership to TS ef76faab133b4cbf96d9e7610be21b62
I20250114 02:52:45.196780 559 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7"
dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
mode: GRACEFUL
new_leader_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:50694
I20250114 02:52:45.197194 559 raft_consensus.cc:604] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 1 LEADER]: Received request to transfer leadership to TS 23a08c9676c449e583b232736f562fad
I20250114 02:52:45.201047 559 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac"
dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
mode: GRACEFUL
new_leader_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:50694
I20250114 02:52:45.201459 559 raft_consensus.cc:604] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 1 LEADER]: Received request to transfer leadership to TS 23a08c9676c449e583b232736f562fad
I20250114 02:52:45.205309 635 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eade2a70e5d54eacb0d30a272d470dfc"
dest_uuid: "23a08c9676c449e583b232736f562fad"
mode: GRACEFUL
new_leader_uuid: "0534eb8430324a1e83c9f26c3d46887e"
from {username='slave'} at 127.0.0.1:50686
I20250114 02:52:45.205703 635 raft_consensus.cc:604] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Received request to transfer leadership to TS 0534eb8430324a1e83c9f26c3d46887e
I20250114 02:52:45.211788 803 raft_consensus.cc:988] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62: : Instructing follower 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:45.212147 797 raft_consensus.cc:1076] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 1 LEADER]: Signalling peer 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:45.213653 635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7"
dest_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:50684
I20250114 02:52:45.214139 635 raft_consensus.cc:491] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:45.214475 635 raft_consensus.cc:3054] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.220011 760 raft_consensus.cc:988] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e: : Instructing follower ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:52:45.220404 771 raft_consensus.cc:1076] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 1 LEADER]: Signalling peer ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:52:45.221062 635 raft_consensus.cc:513] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.221482 559 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "287461283c6247658bdaca7184ac2067"
dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:50680
I20250114 02:52:45.221940 559 raft_consensus.cc:491] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:45.222200 559 raft_consensus.cc:3054] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.223459 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:45.223734 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:45.223971 560 raft_consensus.cc:3049] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:52:45.224310 560 raft_consensus.cc:738] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:52:45.224224 709 raft_consensus.cc:3054] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.225353 560 consensus_queue.cc:260] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.228323 559 raft_consensus.cc:513] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:45.230446 709 raft_consensus.cc:2463] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 2.
I20250114 02:52:45.230934 797 raft_consensus.cc:988] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62: : Instructing follower 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:45.230827 707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287461283c6247658bdaca7184ac2067" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:45.231307 797 raft_consensus.cc:1076] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 1 LEADER]: Signalling peer 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:45.231369 707 raft_consensus.cc:3049] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:52:45.231451 601 leader_election.cc:304] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad; no voters:
I20250114 02:52:45.231840 707 raft_consensus.cc:738] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Running, Role: LEADER
I20250114 02:52:45.226503 560 raft_consensus.cc:3054] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.232316 759 raft_consensus.cc:2798] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:52:45.232806 759 raft_consensus.cc:695] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 2 LEADER]: Becoming Leader. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:45.232609 707 consensus_queue.cc:260] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:45.233697 707 raft_consensus.cc:3054] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.233572 759 consensus_queue.cc:237] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.236610 635 leader_election.cc:290] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 2 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:45.238842 634 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac"
dest_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:50684
I20250114 02:52:45.239451 634 raft_consensus.cc:491] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:45.239712 559 leader_election.cc:290] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 2 election: Requested vote from peers 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429), 23a08c9676c449e583b232736f562fad (127.0.106.66:45709)
I20250114 02:52:45.239848 634 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.239848 707 raft_consensus.cc:2463] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 2.
I20250114 02:52:45.241770 526 leader_election.cc:304] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:45.243283 457 catalog_manager.cc:5526] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad reported cstate change: term changed from 1 to 2, leader changed from ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65) to 23a08c9676c449e583b232736f562fad (127.0.106.66). New cstate: current_term: 2 leader_uuid: "23a08c9676c449e583b232736f562fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:45.243767 797 raft_consensus.cc:2798] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:52:45.244411 797 raft_consensus.cc:695] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Becoming Leader. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:52:45.245327 797 consensus_queue.cc:237] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:45.249437 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287461283c6247658bdaca7184ac2067" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:52:45.250427 635 raft_consensus.cc:3054] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.253809 456 catalog_manager.cc:5526] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 reported cstate change: term changed from 1 to 2, leader changed from 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67) to ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65). New cstate: current_term: 2 leader_uuid: "ef76faab133b4cbf96d9e7610be21b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: HEALTHY } } }
I20250114 02:52:45.256176 635 raft_consensus.cc:2463] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 2.
I20250114 02:52:45.247409 634 raft_consensus.cc:513] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.259107 560 raft_consensus.cc:2463] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 2.
I20250114 02:52:45.260196 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:45.260694 560 raft_consensus.cc:3049] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:52:45.260960 560 raft_consensus.cc:738] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:52:45.261670 560 consensus_queue.cc:260] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.262712 560 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.264598 707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:45.265092 707 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.266670 634 leader_election.cc:290] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 2 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:45.267982 560 raft_consensus.cc:2463] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 2.
I20250114 02:52:45.268860 598 leader_election.cc:304] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a08c9676c449e583b232736f562fad, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:45.269472 787 raft_consensus.cc:2798] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:52:45.270329 787 raft_consensus.cc:695] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 2 LEADER]: Becoming Leader. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:45.271067 707 raft_consensus.cc:2463] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 2.
I20250114 02:52:45.271101 787 consensus_queue.cc:237] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.276824 456 catalog_manager.cc:5526] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad reported cstate change: term changed from 1 to 2, leader changed from ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65) to 23a08c9676c449e583b232736f562fad (127.0.106.66). New cstate: current_term: 2 leader_uuid: "23a08c9676c449e583b232736f562fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:45.294535 759 raft_consensus.cc:988] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad: : Instructing follower ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:52:45.294984 787 raft_consensus.cc:1076] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Signalling peer ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:52:45.296192 560 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d"
dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:50688
I20250114 02:52:45.296598 560 raft_consensus.cc:491] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:45.296833 560 raft_consensus.cc:3054] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.300894 560 raft_consensus.cc:513] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.302605 560 leader_election.cc:290] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 2 election: Requested vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:45.303022 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:52:45.303494 707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:45.303632 634 raft_consensus.cc:3049] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:52:45.303965 707 raft_consensus.cc:3054] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.303969 634 raft_consensus.cc:738] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:45.304824 634 consensus_queue.cc:260] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.305800 634 raft_consensus.cc:3054] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.308275 707 raft_consensus.cc:2463] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 2.
I20250114 02:52:45.309077 526 leader_election.cc:304] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:45.309715 797 raft_consensus.cc:2798] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:52:45.310076 797 raft_consensus.cc:695] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Becoming Leader. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:52:45.310949 634 raft_consensus.cc:2463] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 2.
I20250114 02:52:45.310842 797 consensus_queue.cc:237] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.317456 456 catalog_manager.cc:5526] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 reported cstate change: term changed from 1 to 2, leader changed from 23a08c9676c449e583b232736f562fad (127.0.106.66) to ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65). New cstate: current_term: 2 leader_uuid: "ef76faab133b4cbf96d9e7610be21b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:45.342118 787 consensus_queue.cc:1035] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:45.351516 787 consensus_queue.cc:1035] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:52:45.433032 786 raft_consensus.cc:988] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad: : Instructing follower ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:52:45.433382 787 raft_consensus.cc:1076] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Signalling peer ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:52:45.434428 560 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03c3a4647a594874889f28d2460a5566"
dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:50688
I20250114 02:52:45.434839 560 raft_consensus.cc:491] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:45.435082 560 raft_consensus.cc:3054] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.438827 560 raft_consensus.cc:513] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.440191 560 leader_election.cc:290] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 2 election: Requested vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:45.440799 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c3a4647a594874889f28d2460a5566" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:52:45.441192 707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c3a4647a594874889f28d2460a5566" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:45.441305 634 raft_consensus.cc:3049] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:52:45.441607 634 raft_consensus.cc:738] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:45.441613 707 raft_consensus.cc:3054] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.442179 634 consensus_queue.cc:260] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.443037 634 raft_consensus.cc:3054] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.446466 707 raft_consensus.cc:2463] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 2.
I20250114 02:52:45.447156 634 raft_consensus.cc:2463] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 2.
I20250114 02:52:45.447347 526 leader_election.cc:304] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:45.448081 797 raft_consensus.cc:2798] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:52:45.448405 797 raft_consensus.cc:695] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Becoming Leader. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:52:45.448952 797 consensus_queue.cc:237] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.454532 457 catalog_manager.cc:5526] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 reported cstate change: term changed from 1 to 2, leader changed from 23a08c9676c449e583b232736f562fad (127.0.106.66) to ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65). New cstate: current_term: 2 leader_uuid: "ef76faab133b4cbf96d9e7610be21b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:45.560456 760 raft_consensus.cc:988] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e: : Instructing follower ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:52:45.560779 771 raft_consensus.cc:1076] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 1 LEADER]: Signalling peer ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:52:45.561923 560 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "947325e5f22a41ed906165f55077bc67"
dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:50680
I20250114 02:52:45.562311 560 raft_consensus.cc:491] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:45.562536 560 raft_consensus.cc:3054] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.566712 560 raft_consensus.cc:513] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.568445 560 leader_election.cc:290] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 2 election: Requested vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:45.569160 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947325e5f22a41ed906165f55077bc67" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:52:45.569360 707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947325e5f22a41ed906165f55077bc67" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:45.569662 634 raft_consensus.cc:3054] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.569830 707 raft_consensus.cc:3049] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:52:45.570109 707 raft_consensus.cc:738] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Running, Role: LEADER
I20250114 02:52:45.570698 707 consensus_queue.cc:260] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.571540 707 raft_consensus.cc:3054] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.573606 634 raft_consensus.cc:2463] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 2.
I20250114 02:52:45.574314 525 leader_election.cc:304] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a08c9676c449e583b232736f562fad, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:45.574926 797 raft_consensus.cc:2798] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:52:45.575381 707 raft_consensus.cc:2463] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 2.
I20250114 02:52:45.575449 797 raft_consensus.cc:695] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Becoming Leader. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:52:45.576202 797 consensus_queue.cc:237] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.581393 457 catalog_manager.cc:5526] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 reported cstate change: term changed from 1 to 2, leader changed from 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67) to ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65). New cstate: current_term: 2 leader_uuid: "ef76faab133b4cbf96d9e7610be21b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:45.629010 786 raft_consensus.cc:988] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad: : Instructing follower 0534eb8430324a1e83c9f26c3d46887e to start an election
I20250114 02:52:45.629364 787 raft_consensus.cc:1076] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Signalling peer 0534eb8430324a1e83c9f26c3d46887e to start an election
I20250114 02:52:45.630506 707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eade2a70e5d54eacb0d30a272d470dfc"
dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
from {username='slave'} at 127.0.0.1:38710
I20250114 02:52:45.630900 707 raft_consensus.cc:491] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:45.631145 707 raft_consensus.cc:3054] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.635207 707 raft_consensus.cc:513] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.636492 707 leader_election.cc:290] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 2 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 23a08c9676c449e583b232736f562fad (127.0.106.66:45709)
I20250114 02:52:45.637104 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade2a70e5d54eacb0d30a272d470dfc" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:45.637480 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade2a70e5d54eacb0d30a272d470dfc" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:52:45.638140 560 raft_consensus.cc:3054] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.638221 634 raft_consensus.cc:3049] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:52:45.638540 634 raft_consensus.cc:738] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:45.639189 634 consensus_queue.cc:260] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.640002 634 raft_consensus.cc:3054] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:52:45.641943 560 raft_consensus.cc:2463] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 2.
I20250114 02:52:45.642663 672 leader_election.cc:304] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:45.643296 771 raft_consensus.cc:2798] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:52:45.643734 771 raft_consensus.cc:695] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 2 LEADER]: Becoming Leader. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Running, Role: LEADER
I20250114 02:52:45.644551 771 consensus_queue.cc:237] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:45.645056 634 raft_consensus.cc:2463] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 2.
I20250114 02:52:45.650158 457 catalog_manager.cc:5526] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e reported cstate change: term changed from 1 to 2, leader changed from 23a08c9676c449e583b232736f562fad (127.0.106.66) to 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67). New cstate: current_term: 2 leader_uuid: "0534eb8430324a1e83c9f26c3d46887e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: HEALTHY } } }
I20250114 02:52:45.712577 707 raft_consensus.cc:1270] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:45.713644 797 consensus_queue.cc:1035] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:45.721349 634 raft_consensus.cc:1270] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:45.723383 758 consensus_queue.cc:1035] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:45.731561 707 raft_consensus.cc:1270] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:45.732641 787 consensus_queue.cc:1035] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:45.740692 560 raft_consensus.cc:1270] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:45.741711 786 consensus_queue.cc:1035] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:45.781793 707 raft_consensus.cc:1270] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:45.782891 786 consensus_queue.cc:1035] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:45.790383 560 raft_consensus.cc:1270] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:45.791620 786 consensus_queue.cc:1035] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:45.848726 634 raft_consensus.cc:1270] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:45.849849 758 consensus_queue.cc:1035] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:45.858501 707 raft_consensus.cc:1270] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:45.859732 803 consensus_queue.cc:1035] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:45.952590 707 raft_consensus.cc:1270] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:45.953672 803 consensus_queue.cc:1035] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:45.960827 634 raft_consensus.cc:1270] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:45.962278 797 consensus_queue.cc:1035] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:45.998474 707 raft_consensus.cc:1270] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:45.999542 797 consensus_queue.cc:1035] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:46.006361 634 raft_consensus.cc:1270] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:46.008219 803 consensus_queue.cc:1035] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:46.120888 560 raft_consensus.cc:1270] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Refusing update from remote peer 0534eb8430324a1e83c9f26c3d46887e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:46.121963 771 consensus_queue.cc:1035] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:46.129698 634 raft_consensus.cc:1270] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Refusing update from remote peer 0534eb8430324a1e83c9f26c3d46887e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:52:46.131778 760 consensus_queue.cc:1035] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:52:48.228447 425 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20250114 02:52:48.229137 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:52:48.242146 560 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03c3a4647a594874889f28d2460a5566"
dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
mode: GRACEFUL
new_leader_uuid: "0534eb8430324a1e83c9f26c3d46887e"
from {username='slave'} at 127.0.0.1:50708
I20250114 02:52:48.242658 560 raft_consensus.cc:604] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Received request to transfer leadership to TS 0534eb8430324a1e83c9f26c3d46887e
I20250114 02:52:48.252652 634 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac"
dest_uuid: "23a08c9676c449e583b232736f562fad"
mode: GRACEFUL
new_leader_uuid: "0534eb8430324a1e83c9f26c3d46887e"
from {username='slave'} at 127.0.0.1:50700
I20250114 02:52:48.253075 634 raft_consensus.cc:604] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 2 LEADER]: Received request to transfer leadership to TS 0534eb8430324a1e83c9f26c3d46887e
I20250114 02:52:48.253926 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250114 02:52:48.254314 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:52:48.366578 856 raft_consensus.cc:988] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62: : Instructing follower 0534eb8430324a1e83c9f26c3d46887e to start an election
I20250114 02:52:48.367063 803 raft_consensus.cc:1076] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Signalling peer 0534eb8430324a1e83c9f26c3d46887e to start an election
I20250114 02:52:48.368372 707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03c3a4647a594874889f28d2460a5566"
dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
from {username='slave'} at 127.0.0.1:38722
I20250114 02:52:48.368803 707 raft_consensus.cc:491] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:48.369138 707 raft_consensus.cc:3054] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:48.375679 707 raft_consensus.cc:513] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:48.378055 707 leader_election.cc:290] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 3 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 23a08c9676c449e583b232736f562fad (127.0.106.66:45709)
I20250114 02:52:48.378608 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c3a4647a594874889f28d2460a5566" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:48.379134 560 raft_consensus.cc:3049] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:52:48.379334 560 raft_consensus.cc:738] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:52:48.379861 560 consensus_queue.cc:260] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:48.380682 560 raft_consensus.cc:3054] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:48.383172 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c3a4647a594874889f28d2460a5566" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:52:48.383705 634 raft_consensus.cc:3054] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:48.385514 560 raft_consensus.cc:2463] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 3.
I20250114 02:52:48.386528 672 leader_election.cc:304] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:48.387226 854 raft_consensus.cc:2798] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:52:48.388381 854 raft_consensus.cc:695] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 3 LEADER]: Becoming Leader. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Running, Role: LEADER
I20250114 02:52:48.388651 634 raft_consensus.cc:2463] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 3.
I20250114 02:52:48.389135 854 consensus_queue.cc:237] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:48.396955 457 catalog_manager.cc:5526] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e reported cstate change: term changed from 2 to 3, leader changed from ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65) to 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67). New cstate: current_term: 3 leader_uuid: "0534eb8430324a1e83c9f26c3d46887e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: HEALTHY } } }
I20250114 02:52:48.541479 855 raft_consensus.cc:988] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad: : Instructing follower 0534eb8430324a1e83c9f26c3d46887e to start an election
I20250114 02:52:48.541828 866 raft_consensus.cc:1076] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 2 LEADER]: Signalling peer 0534eb8430324a1e83c9f26c3d46887e to start an election
I20250114 02:52:48.542979 707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac"
dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
from {username='slave'} at 127.0.0.1:38710
I20250114 02:52:48.543339 707 raft_consensus.cc:491] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:48.543565 707 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:48.547428 707 raft_consensus.cc:513] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:48.548849 707 leader_election.cc:290] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 3 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 23a08c9676c449e583b232736f562fad (127.0.106.66:45709)
I20250114 02:52:48.549479 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:48.549762 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:52:48.550065 560 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:48.550271 634 raft_consensus.cc:3049] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:52:48.550544 634 raft_consensus.cc:738] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:48.551198 634 consensus_queue.cc:260] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:48.552224 634 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:48.555497 560 raft_consensus.cc:2463] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 3.
I20250114 02:52:48.556393 672 leader_election.cc:304] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:48.556955 854 raft_consensus.cc:2798] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:52:48.557281 854 raft_consensus.cc:695] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 3 LEADER]: Becoming Leader. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Running, Role: LEADER
I20250114 02:52:48.557374 634 raft_consensus.cc:2463] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 3.
I20250114 02:52:48.557896 854 consensus_queue.cc:237] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:48.563830 457 catalog_manager.cc:5526] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e reported cstate change: term changed from 2 to 3, leader changed from 23a08c9676c449e583b232736f562fad (127.0.106.66) to 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67). New cstate: current_term: 3 leader_uuid: "0534eb8430324a1e83c9f26c3d46887e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: HEALTHY } } }
I20250114 02:52:48.936856 634 raft_consensus.cc:1270] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Refusing update from remote peer 0534eb8430324a1e83c9f26c3d46887e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:52:48.938063 854 consensus_queue.cc:1035] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:52:48.946108 560 raft_consensus.cc:1270] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Refusing update from remote peer 0534eb8430324a1e83c9f26c3d46887e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:52:48.947692 854 consensus_queue.cc:1035] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:52:49.010818 634 raft_consensus.cc:1270] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Refusing update from remote peer 0534eb8430324a1e83c9f26c3d46887e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:52:49.011857 854 consensus_queue.cc:1035] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:52:49.016793 560 raft_consensus.cc:1270] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Refusing update from remote peer 0534eb8430324a1e83c9f26c3d46887e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:52:49.018234 860 consensus_queue.cc:1035] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:52:49.255082 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:52:49.259896 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:52:49.260210 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:52:50.260879 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:52:50.265354 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:52:50.265668 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:52:51.266443 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:52:51.271700 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:52:51.271991 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:52:52.272770 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:52:52.277256 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:52:52.277539 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:52:53.278118 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:52:53.282732 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:52:53.283031 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:52:54.283694 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:52:54.287990 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:52:54.288296 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:52:55.289004 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:52:55.293457 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:52:55.293771 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:52:56.294433 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:52:56.298933 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:52:56.299240 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:52:57.299877 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:52:57.303841 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:52:57.304100 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:52:58.325685 425 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20250114 02:52:58.329540 560 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "287461283c6247658bdaca7184ac2067"
dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
mode: GRACEFUL
new_leader_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:50694
I20250114 02:52:58.330138 560 raft_consensus.cc:604] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Received request to transfer leadership to TS 23a08c9676c449e583b232736f562fad
I20250114 02:52:58.333894 560 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "947325e5f22a41ed906165f55077bc67"
dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
mode: GRACEFUL
new_leader_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:50694
I20250114 02:52:58.334249 560 raft_consensus.cc:604] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Received request to transfer leadership to TS 23a08c9676c449e583b232736f562fad
I20250114 02:52:58.338102 560 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d"
dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
mode: GRACEFUL
new_leader_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:50694
I20250114 02:52:58.338464 560 raft_consensus.cc:604] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Received request to transfer leadership to TS 23a08c9676c449e583b232736f562fad
I20250114 02:52:58.342006 707 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03c3a4647a594874889f28d2460a5566"
dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
mode: GRACEFUL
new_leader_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:38728
I20250114 02:52:58.342353 707 raft_consensus.cc:604] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 3 LEADER]: Received request to transfer leadership to TS 23a08c9676c449e583b232736f562fad
I20250114 02:52:58.351504 707 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac"
dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
mode: GRACEFUL
new_leader_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:38728
I20250114 02:52:58.351823 707 raft_consensus.cc:604] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 3 LEADER]: Received request to transfer leadership to TS 23a08c9676c449e583b232736f562fad
I20250114 02:52:58.417296 887 raft_consensus.cc:988] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e: : Instructing follower 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:58.417979 882 raft_consensus.cc:1076] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 3 LEADER]: Signalling peer 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:58.419734 635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03c3a4647a594874889f28d2460a5566"
dest_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:50674
I20250114 02:52:58.420426 635 raft_consensus.cc:491] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:58.420950 635 raft_consensus.cc:3054] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:52:58.430194 635 raft_consensus.cc:513] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:58.432437 635 leader_election.cc:290] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 4 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:58.433357 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c3a4647a594874889f28d2460a5566" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:58.434019 560 raft_consensus.cc:3054] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:52:58.434962 707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c3a4647a594874889f28d2460a5566" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:58.435688 707 raft_consensus.cc:3049] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:52:58.436115 707 raft_consensus.cc:738] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Running, Role: LEADER
I20250114 02:52:58.437100 707 consensus_queue.cc:260] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:58.438549 707 raft_consensus.cc:3054] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:52:58.439743 560 raft_consensus.cc:2463] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 4.
I20250114 02:52:58.441406 598 leader_election.cc:304] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a08c9676c449e583b232736f562fad, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:58.442368 886 raft_consensus.cc:2798] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:52:58.442831 886 raft_consensus.cc:695] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 4 LEADER]: Becoming Leader. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:58.443786 886 consensus_queue.cc:237] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:58.444312 707 raft_consensus.cc:2463] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 4.
I20250114 02:52:58.451026 457 catalog_manager.cc:5526] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad reported cstate change: term changed from 3 to 4, leader changed from 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67) to 23a08c9676c449e583b232736f562fad (127.0.106.66). New cstate: current_term: 4 leader_uuid: "23a08c9676c449e583b232736f562fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:58.555145 883 raft_consensus.cc:988] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62: : Instructing follower 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:58.555583 893 raft_consensus.cc:1076] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Signalling peer 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:58.556695 635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "287461283c6247658bdaca7184ac2067"
dest_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:50684
I20250114 02:52:58.557054 635 raft_consensus.cc:491] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:58.557253 635 raft_consensus.cc:3054] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:58.561324 635 raft_consensus.cc:513] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:58.562608 635 leader_election.cc:290] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 3 election: Requested vote from peers 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429), ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183)
I20250114 02:52:58.563285 707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287461283c6247658bdaca7184ac2067" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:58.563452 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287461283c6247658bdaca7184ac2067" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:58.563738 707 raft_consensus.cc:3054] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:58.563851 560 raft_consensus.cc:3049] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:52:58.564133 560 raft_consensus.cc:738] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:52:58.564635 560 consensus_queue.cc:260] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:58.565493 560 raft_consensus.cc:3054] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:58.567564 707 raft_consensus.cc:2463] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 3.
I20250114 02:52:58.568604 601 leader_election.cc:304] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad; no voters:
I20250114 02:52:58.569353 886 raft_consensus.cc:2798] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:52:58.569701 886 raft_consensus.cc:695] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 3 LEADER]: Becoming Leader. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:58.569757 560 raft_consensus.cc:2463] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 3.
I20250114 02:52:58.570576 886 consensus_queue.cc:237] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:52:58.576844 457 catalog_manager.cc:5526] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad reported cstate change: term changed from 2 to 3, leader changed from ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65) to 23a08c9676c449e583b232736f562fad (127.0.106.66). New cstate: current_term: 3 leader_uuid: "23a08c9676c449e583b232736f562fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:58.693341 887 raft_consensus.cc:988] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e: : Instructing follower 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:58.693702 882 raft_consensus.cc:1076] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 3 LEADER]: Signalling peer 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:58.694984 635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac"
dest_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:50674
I20250114 02:52:58.695411 635 raft_consensus.cc:491] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:58.695664 635 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:52:58.700153 635 raft_consensus.cc:513] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:58.701432 635 leader_election.cc:290] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 4 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:58.702103 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:58.702389 707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:58.702668 560 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:52:58.702886 707 raft_consensus.cc:3049] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:52:58.703161 707 raft_consensus.cc:738] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Running, Role: LEADER
I20250114 02:52:58.703796 707 consensus_queue.cc:260] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:58.704802 707 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:52:58.704869 883 raft_consensus.cc:988] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62: : Instructing follower 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:58.705439 893 raft_consensus.cc:1076] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Signalling peer 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:58.706897 635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d"
dest_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:50684
I20250114 02:52:58.707446 635 raft_consensus.cc:491] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:58.707855 635 raft_consensus.cc:3054] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:58.708204 560 raft_consensus.cc:2463] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 4.
I20250114 02:52:58.709326 598 leader_election.cc:304] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a08c9676c449e583b232736f562fad, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:58.710129 886 raft_consensus.cc:2798] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:52:58.710572 886 raft_consensus.cc:695] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 4 LEADER]: Becoming Leader. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:58.710970 707 raft_consensus.cc:2463] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 4.
I20250114 02:52:58.711519 886 consensus_queue.cc:237] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:58.714685 883 raft_consensus.cc:988] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62: : Instructing follower 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:58.714529 635 raft_consensus.cc:513] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:58.715126 883 raft_consensus.cc:1076] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Signalling peer 23a08c9676c449e583b232736f562fad to start an election
I20250114 02:52:58.716912 634 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "947325e5f22a41ed906165f55077bc67"
dest_uuid: "23a08c9676c449e583b232736f562fad"
from {username='slave'} at 127.0.0.1:50684
I20250114 02:52:58.717317 635 leader_election.cc:290] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 3 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:58.717424 634 raft_consensus.cc:491] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:52:58.717474 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:58.717769 634 raft_consensus.cc:3054] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:58.718007 560 raft_consensus.cc:3049] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:52:58.718307 560 raft_consensus.cc:738] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:52:58.718969 560 consensus_queue.cc:260] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:58.719841 560 raft_consensus.cc:3054] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:58.721194 457 catalog_manager.cc:5526] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad reported cstate change: term changed from 3 to 4, leader changed from 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67) to 23a08c9676c449e583b232736f562fad (127.0.106.66). New cstate: current_term: 4 leader_uuid: "23a08c9676c449e583b232736f562fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:58.721805 634 raft_consensus.cc:513] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:58.723605 707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:58.724121 707 raft_consensus.cc:3054] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:58.725080 634 leader_election.cc:290] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 3 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:52:58.725145 560 raft_consensus.cc:2463] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 3.
I20250114 02:52:58.726156 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947325e5f22a41ed906165f55077bc67" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:52:58.726730 709 raft_consensus.cc:3054] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:58.727015 598 leader_election.cc:304] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a08c9676c449e583b232736f562fad, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:52:58.727849 886 raft_consensus.cc:2798] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:52:58.728363 886 raft_consensus.cc:695] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 3 LEADER]: Becoming Leader. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:58.727941 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947325e5f22a41ed906165f55077bc67" candidate_uuid: "23a08c9676c449e583b232736f562fad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:52:58.729267 560 raft_consensus.cc:3049] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:52:58.729601 560 raft_consensus.cc:738] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:52:58.729177 886 consensus_queue.cc:237] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:58.729681 707 raft_consensus.cc:2463] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 3.
I20250114 02:52:58.730309 560 consensus_queue.cc:260] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:58.731429 560 raft_consensus.cc:3054] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:52:58.735110 709 raft_consensus.cc:2463] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 3.
I20250114 02:52:58.736249 601 leader_election.cc:304] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, 23a08c9676c449e583b232736f562fad; no voters:
I20250114 02:52:58.736830 560 raft_consensus.cc:2463] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a08c9676c449e583b232736f562fad in term 3.
I20250114 02:52:58.737010 866 raft_consensus.cc:2798] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:52:58.737457 866 raft_consensus.cc:695] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 3 LEADER]: Becoming Leader. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:52:58.738248 866 consensus_queue.cc:237] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:52:58.739385 457 catalog_manager.cc:5526] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad reported cstate change: term changed from 2 to 3, leader changed from ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65) to 23a08c9676c449e583b232736f562fad (127.0.106.66). New cstate: current_term: 3 leader_uuid: "23a08c9676c449e583b232736f562fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:58.749389 457 catalog_manager.cc:5526] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad reported cstate change: term changed from 2 to 3, leader changed from ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65) to 23a08c9676c449e583b232736f562fad (127.0.106.66). New cstate: current_term: 3 leader_uuid: "23a08c9676c449e583b232736f562fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:52:58.979331 709 raft_consensus.cc:1270] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:52:58.980299 866 consensus_queue.cc:1035] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:52:58.991127 560 raft_consensus.cc:1270] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:52:58.992408 866 consensus_queue.cc:1035] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:52:59.096885 709 raft_consensus.cc:1270] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:52:59.097877 886 consensus_queue.cc:1035] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:52:59.105150 560 raft_consensus.cc:1270] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:52:59.106479 886 consensus_queue.cc:1035] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:52:59.120672 709 raft_consensus.cc:1270] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:52:59.121618 886 consensus_queue.cc:1035] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:52:59.125643 707 raft_consensus.cc:1270] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:52:59.126739 886 consensus_queue.cc:1035] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:52:59.129601 560 raft_consensus.cc:1270] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:52:59.131234 901 consensus_queue.cc:1035] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:52:59.132474 560 raft_consensus.cc:1270] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:52:59.134861 901 consensus_queue.cc:1035] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:52:59.250396 709 raft_consensus.cc:1270] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:52:59.251523 901 consensus_queue.cc:1035] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:52:59.259495 560 raft_consensus.cc:1270] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Refusing update from remote peer 23a08c9676c449e583b232736f562fad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:52:59.260850 911 consensus_queue.cc:1035] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:53:01.379954 425 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20250114 02:53:01.380806 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:01.386974 634 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac"
dest_uuid: "23a08c9676c449e583b232736f562fad"
mode: GRACEFUL
new_leader_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:50700
I20250114 02:53:01.387542 634 raft_consensus.cc:604] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 4 LEADER]: Received request to transfer leadership to TS ef76faab133b4cbf96d9e7610be21b62
I20250114 02:53:01.389634 634 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "287461283c6247658bdaca7184ac2067"
dest_uuid: "23a08c9676c449e583b232736f562fad"
mode: GRACEFUL
new_leader_uuid: "0534eb8430324a1e83c9f26c3d46887e"
from {username='slave'} at 127.0.0.1:50700
I20250114 02:53:01.389974 634 raft_consensus.cc:604] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 3 LEADER]: Received request to transfer leadership to TS 0534eb8430324a1e83c9f26c3d46887e
I20250114 02:53:01.391786 634 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d"
dest_uuid: "23a08c9676c449e583b232736f562fad"
mode: GRACEFUL
new_leader_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:50700
I20250114 02:53:01.392140 634 raft_consensus.cc:604] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 3 LEADER]: Received request to transfer leadership to TS ef76faab133b4cbf96d9e7610be21b62
I20250114 02:53:01.393793 634 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6acde705ff2a429985e29c3cb68c983d"
dest_uuid: "23a08c9676c449e583b232736f562fad"
mode: GRACEFUL
new_leader_uuid: "0534eb8430324a1e83c9f26c3d46887e"
from {username='slave'} at 127.0.0.1:50700
I20250114 02:53:01.394125 634 raft_consensus.cc:604] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Received request to transfer leadership to TS 0534eb8430324a1e83c9f26c3d46887e
I20250114 02:53:01.395903 634 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7"
dest_uuid: "23a08c9676c449e583b232736f562fad"
mode: GRACEFUL
new_leader_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:50700
I20250114 02:53:01.396210 634 raft_consensus.cc:604] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 2 LEADER]: Received request to transfer leadership to TS ef76faab133b4cbf96d9e7610be21b62
I20250114 02:53:01.397024 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250114 02:53:01.397395 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:01.626824 915 raft_consensus.cc:988] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad: : Instructing follower ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:53:01.627323 911 raft_consensus.cc:1076] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 3 LEADER]: Signalling peer ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:53:01.628580 560 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d"
dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:50688
I20250114 02:53:01.628959 560 raft_consensus.cc:491] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:01.629259 560 raft_consensus.cc:3054] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:01.634021 560 raft_consensus.cc:513] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:53:01.635919 560 leader_election.cc:290] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 4 election: Requested vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:53:01.636721 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:53:01.636989 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8a0de188ee481e8b79fd40d711382d" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:53:01.637413 634 raft_consensus.cc:3049] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:53:01.637462 709 raft_consensus.cc:3054] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:01.637743 634 raft_consensus.cc:738] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:53:01.638358 634 consensus_queue.cc:260] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:53:01.639322 634 raft_consensus.cc:3054] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:01.642522 709 raft_consensus.cc:2463] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 4.
I20250114 02:53:01.643630 526 leader_election.cc:304] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:53:01.644817 634 raft_consensus.cc:2463] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 4.
I20250114 02:53:01.645217 916 raft_consensus.cc:2798] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:53:01.646291 916 raft_consensus.cc:695] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 4 LEADER]: Becoming Leader. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:53:01.647114 916 consensus_queue.cc:237] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:53:01.653496 455 catalog_manager.cc:5526] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 reported cstate change: term changed from 3 to 4, leader changed from 23a08c9676c449e583b232736f562fad (127.0.106.66) to ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65). New cstate: current_term: 4 leader_uuid: "ef76faab133b4cbf96d9e7610be21b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:01.743440 915 raft_consensus.cc:988] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad: : Instructing follower ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:53:01.743865 911 raft_consensus.cc:1076] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 2 LEADER]: Signalling peer ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:53:01.745034 560 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7"
dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:50688
I20250114 02:53:01.745398 560 raft_consensus.cc:491] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:01.745616 560 raft_consensus.cc:3054] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:01.750432 560 raft_consensus.cc:513] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:53:01.751675 560 leader_election.cc:290] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 3 election: Requested vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:53:01.752352 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:53:01.752571 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a601291dd147628db8399e31b46fe7" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:53:01.752856 634 raft_consensus.cc:3049] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:01.753021 709 raft_consensus.cc:3054] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:01.753135 634 raft_consensus.cc:738] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:53:01.753772 634 consensus_queue.cc:260] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:53:01.754608 634 raft_consensus.cc:3054] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:01.757144 709 raft_consensus.cc:2463] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 3.
I20250114 02:53:01.757905 526 leader_election.cc:304] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:53:01.758622 916 raft_consensus.cc:2798] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:01.758941 634 raft_consensus.cc:2463] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 3.
I20250114 02:53:01.758978 916 raft_consensus.cc:695] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 3 LEADER]: Becoming Leader. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:53:01.759907 916 consensus_queue.cc:237] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:53:01.765082 455 catalog_manager.cc:5526] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 reported cstate change: term changed from 2 to 3, leader changed from 23a08c9676c449e583b232736f562fad (127.0.106.66) to ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65). New cstate: current_term: 3 leader_uuid: "ef76faab133b4cbf96d9e7610be21b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:01.777302 915 raft_consensus.cc:988] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad: : Instructing follower ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:53:01.777670 911 raft_consensus.cc:1076] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 4 LEADER]: Signalling peer ef76faab133b4cbf96d9e7610be21b62 to start an election
I20250114 02:53:01.778748 560 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac"
dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
from {username='slave'} at 127.0.0.1:50688
I20250114 02:53:01.779107 560 raft_consensus.cc:491] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:01.779309 560 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:53:01.783044 560 raft_consensus.cc:513] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:53:01.784474 560 leader_election.cc:290] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 5 election: Requested vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67:38429)
I20250114 02:53:01.784950 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:53:01.785306 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac89c2972b74708b89db61fb9f846ac" candidate_uuid: "ef76faab133b4cbf96d9e7610be21b62" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
I20250114 02:53:01.785441 634 raft_consensus.cc:3049] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:53:01.785781 709 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:53:01.785791 634 raft_consensus.cc:738] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:53:01.786532 634 consensus_queue.cc:260] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:53:01.787638 634 raft_consensus.cc:3054] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:53:01.791143 709 raft_consensus.cc:2463] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 5.
I20250114 02:53:01.791954 526 leader_election.cc:304] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:53:01.792527 916 raft_consensus.cc:2798] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:53:01.792717 634 raft_consensus.cc:2463] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef76faab133b4cbf96d9e7610be21b62 in term 5.
I20250114 02:53:01.792965 916 raft_consensus.cc:695] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 5 LEADER]: Becoming Leader. State: Replica: ef76faab133b4cbf96d9e7610be21b62, State: Running, Role: LEADER
I20250114 02:53:01.793854 916 consensus_queue.cc:237] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:53:01.799774 455 catalog_manager.cc:5526] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 reported cstate change: term changed from 4 to 5, leader changed from 23a08c9676c449e583b232736f562fad (127.0.106.66) to ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65). New cstate: current_term: 5 leader_uuid: "ef76faab133b4cbf96d9e7610be21b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:01.804648 915 raft_consensus.cc:988] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad: : Instructing follower 0534eb8430324a1e83c9f26c3d46887e to start an election
I20250114 02:53:01.805047 911 raft_consensus.cc:1076] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Signalling peer 0534eb8430324a1e83c9f26c3d46887e to start an election
I20250114 02:53:01.806183 709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6acde705ff2a429985e29c3cb68c983d"
dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
from {username='slave'} at 127.0.0.1:38710
I20250114 02:53:01.806529 709 raft_consensus.cc:491] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:01.806824 709 raft_consensus.cc:3054] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:01.810470 709 raft_consensus.cc:513] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:53:01.812037 709 leader_election.cc:290] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 2 election: Requested vote from peers ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183), 23a08c9676c449e583b232736f562fad (127.0.106.66:45709)
I20250114 02:53:01.812740 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acde705ff2a429985e29c3cb68c983d" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:53:01.812793 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acde705ff2a429985e29c3cb68c983d" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:53:01.813318 560 raft_consensus.cc:3054] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:01.813388 634 raft_consensus.cc:3049] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:01.813637 634 raft_consensus.cc:738] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:53:01.814163 634 consensus_queue.cc:260] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:53:01.814931 634 raft_consensus.cc:3054] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:01.816777 560 raft_consensus.cc:2463] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 2.
I20250114 02:53:01.817751 672 leader_election.cc:304] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:53:01.818607 882 raft_consensus.cc:2798] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:01.818801 634 raft_consensus.cc:2463] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 2.
I20250114 02:53:01.819824 882 raft_consensus.cc:695] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 2 LEADER]: Becoming Leader. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Running, Role: LEADER
I20250114 02:53:01.820577 882 consensus_queue.cc:237] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } }
I20250114 02:53:01.826027 455 catalog_manager.cc:5526] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e reported cstate change: term changed from 1 to 2, leader changed from 23a08c9676c449e583b232736f562fad (127.0.106.66) to 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67). New cstate: current_term: 2 leader_uuid: "0534eb8430324a1e83c9f26c3d46887e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:01.840507 915 raft_consensus.cc:988] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad: : Instructing follower 0534eb8430324a1e83c9f26c3d46887e to start an election
I20250114 02:53:01.840896 915 raft_consensus.cc:1076] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 3 LEADER]: Signalling peer 0534eb8430324a1e83c9f26c3d46887e to start an election
I20250114 02:53:01.842064 709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "287461283c6247658bdaca7184ac2067"
dest_uuid: "0534eb8430324a1e83c9f26c3d46887e"
from {username='slave'} at 127.0.0.1:38710
I20250114 02:53:01.842427 709 raft_consensus.cc:491] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:01.842832 709 raft_consensus.cc:3054] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:01.849185 709 raft_consensus.cc:513] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:53:01.850915 709 leader_election.cc:290] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 4 election: Requested vote from peers 23a08c9676c449e583b232736f562fad (127.0.106.66:45709), ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183)
I20250114 02:53:01.852031 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287461283c6247658bdaca7184ac2067" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "23a08c9676c449e583b232736f562fad"
I20250114 02:53:01.852119 560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287461283c6247658bdaca7184ac2067" candidate_uuid: "0534eb8430324a1e83c9f26c3d46887e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ef76faab133b4cbf96d9e7610be21b62"
I20250114 02:53:01.852455 634 raft_consensus.cc:3049] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:53:01.852600 560 raft_consensus.cc:3054] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:01.852711 634 raft_consensus.cc:738] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 23a08c9676c449e583b232736f562fad, State: Running, Role: LEADER
I20250114 02:53:01.853238 634 consensus_queue.cc:260] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:53:01.853919 634 raft_consensus.cc:3054] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:01.856360 560 raft_consensus.cc:2463] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 4.
I20250114 02:53:01.857040 672 leader_election.cc:304] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0534eb8430324a1e83c9f26c3d46887e, ef76faab133b4cbf96d9e7610be21b62; no voters:
I20250114 02:53:01.857578 882 raft_consensus.cc:2798] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:53:01.857738 634 raft_consensus.cc:2463] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0534eb8430324a1e83c9f26c3d46887e in term 4.
I20250114 02:53:01.857882 882 raft_consensus.cc:695] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 4 LEADER]: Becoming Leader. State: Replica: 0534eb8430324a1e83c9f26c3d46887e, State: Running, Role: LEADER
I20250114 02:53:01.858870 882 consensus_queue.cc:237] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } }
I20250114 02:53:01.864683 455 catalog_manager.cc:5526] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e reported cstate change: term changed from 3 to 4, leader changed from 23a08c9676c449e583b232736f562fad (127.0.106.66) to 0534eb8430324a1e83c9f26c3d46887e (127.0.106.67). New cstate: current_term: 4 leader_uuid: "0534eb8430324a1e83c9f26c3d46887e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:02.082592 634 raft_consensus.cc:1270] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:53:02.083585 916 consensus_queue.cc:1035] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:53:02.093191 709 raft_consensus.cc:1270] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:53:02.095217 916 consensus_queue.cc:1035] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:53:02.244390 560 raft_consensus.cc:1270] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Refusing update from remote peer 0534eb8430324a1e83c9f26c3d46887e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:02.245427 882 consensus_queue.cc:1035] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:02.253661 634 raft_consensus.cc:1270] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Refusing update from remote peer 0534eb8430324a1e83c9f26c3d46887e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:02.255106 934 consensus_queue.cc:1035] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:53:02.256736 634 raft_consensus.cc:1270] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:53:02.258344 917 consensus_queue.cc:1035] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 02:53:02.273910 709 raft_consensus.cc:1270] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:53:02.275077 916 consensus_queue.cc:1035] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 02:53:02.330735 709 raft_consensus.cc:1270] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 5 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250114 02:53:02.331760 917 consensus_queue.cc:1035] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534eb8430324a1e83c9f26c3d46887e" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 02:53:02.338018 634 raft_consensus.cc:1270] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 5 FOLLOWER]: Refusing update from remote peer ef76faab133b4cbf96d9e7610be21b62: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250114 02:53:02.339406 917 consensus_queue.cc:1035] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 02:53:02.351465 560 raft_consensus.cc:1270] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Refusing update from remote peer 0534eb8430324a1e83c9f26c3d46887e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:53:02.352555 934 consensus_queue.cc:1035] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef76faab133b4cbf96d9e7610be21b62" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:53:02.361078 634 raft_consensus.cc:1270] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Refusing update from remote peer 0534eb8430324a1e83c9f26c3d46887e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:53:02.362365 882 consensus_queue.cc:1035] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a08c9676c449e583b232736f562fad" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:53:02.398125 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:02.402935 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:02.403267 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:03.403944 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:03.408396 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:03.408746 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:04.409467 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:04.413959 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:04.414327 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:05.415025 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:05.419842 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:05.420225 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:06.420907 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:06.425523 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:06.425871 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:07.426602 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:07.431162 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:07.431514 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:08.432296 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:08.436992 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:08.437382 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:09.438076 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:09.442705 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:09.443037 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:10.443738 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:10.448529 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:10.448868 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:11.474191 425 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20250114 02:53:11.474795 425 tablet_server.cc:178] TabletServer@127.0.106.65:0 shutting down...
I20250114 02:53:11.499176 425 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 02:53:11.500008 425 tablet_replica.cc:331] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62: stopping tablet replica
I20250114 02:53:11.500676 425 raft_consensus.cc:2238] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.501206 425 raft_consensus.cc:2267] T 6acde705ff2a429985e29c3cb68c983d P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.505475 425 tablet_replica.cc:331] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62: stopping tablet replica
I20250114 02:53:11.505952 425 raft_consensus.cc:2238] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.506347 425 raft_consensus.cc:2267] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P ef76faab133b4cbf96d9e7610be21b62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.507949 425 tablet_replica.cc:331] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62: stopping tablet replica
I20250114 02:53:11.508404 425 raft_consensus.cc:2238] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 5 LEADER]: Raft consensus shutting down.
I20250114 02:53:11.509166 425 raft_consensus.cc:2267] T 0ac89c2972b74708b89db61fb9f846ac P ef76faab133b4cbf96d9e7610be21b62 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.510733 425 tablet_replica.cc:331] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62: stopping tablet replica
I20250114 02:53:11.511196 425 raft_consensus.cc:2238] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 3 LEADER]: Raft consensus shutting down.
I20250114 02:53:11.511878 425 raft_consensus.cc:2267] T c2a601291dd147628db8399e31b46fe7 P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.513485 425 tablet_replica.cc:331] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62: stopping tablet replica
I20250114 02:53:11.513919 425 raft_consensus.cc:2238] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.514274 425 raft_consensus.cc:2267] T 287461283c6247658bdaca7184ac2067 P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.515874 425 tablet_replica.cc:331] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62: stopping tablet replica
I20250114 02:53:11.516279 425 raft_consensus.cc:2238] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.516636 425 raft_consensus.cc:2267] T eade2a70e5d54eacb0d30a272d470dfc P ef76faab133b4cbf96d9e7610be21b62 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.518042 425 tablet_replica.cc:331] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62: stopping tablet replica
I20250114 02:53:11.518448 425 raft_consensus.cc:2238] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.518875 425 raft_consensus.cc:2267] T 947325e5f22a41ed906165f55077bc67 P ef76faab133b4cbf96d9e7610be21b62 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.520382 425 tablet_replica.cc:331] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62: stopping tablet replica
I20250114 02:53:11.520766 425 raft_consensus.cc:2238] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.521221 425 raft_consensus.cc:2267] T 03c3a4647a594874889f28d2460a5566 P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.522712 425 tablet_replica.cc:331] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62: stopping tablet replica
I20250114 02:53:11.523314 425 raft_consensus.cc:2238] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 4 LEADER]: Raft consensus shutting down.
I20250114 02:53:11.524284 425 raft_consensus.cc:2267] T 2b8a0de188ee481e8b79fd40d711382d P ef76faab133b4cbf96d9e7610be21b62 [term 4 FOLLOWER]: Raft consensus is shut down!
W20250114 02:53:11.610338 601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:38429, user_credentials={real_user=slave}} blocked reactor thread for 62613.3us
W20250114 02:53:11.611665 598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.65:36183: connect: Connection refused (error 111)
I20250114 02:53:11.616075 425 tablet_server.cc:195] TabletServer@127.0.106.65:0 shutdown complete.
W20250114 02:53:11.616163 598 consensus_peers.cc:487] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad -> Peer ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183): Couldn't send request to peer ef76faab133b4cbf96d9e7610be21b62. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.65:36183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:53:11.623747 598 consensus_peers.cc:487] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad -> Peer ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183): Couldn't send request to peer ef76faab133b4cbf96d9e7610be21b62. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.65:36183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:53:11.625635 598 consensus_peers.cc:487] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad -> Peer ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183): Couldn't send request to peer ef76faab133b4cbf96d9e7610be21b62. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.65:36183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:11.638643 425 tablet_server.cc:178] TabletServer@127.0.106.66:0 shutting down...
W20250114 02:53:11.662266 672 consensus_peers.cc:487] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e -> Peer ef76faab133b4cbf96d9e7610be21b62 (127.0.106.65:36183): Couldn't send request to peer ef76faab133b4cbf96d9e7610be21b62. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.65:36183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:11.668988 425 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 02:53:11.669603 425 tablet_replica.cc:331] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad: stopping tablet replica
I20250114 02:53:11.670130 425 raft_consensus.cc:2238] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.670532 425 raft_consensus.cc:2267] T 6acde705ff2a429985e29c3cb68c983d P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.672284 425 tablet_replica.cc:331] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad: stopping tablet replica
I20250114 02:53:11.672760 425 raft_consensus.cc:2238] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.673112 425 raft_consensus.cc:2267] T c2a601291dd147628db8399e31b46fe7 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.674536 425 tablet_replica.cc:331] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad: stopping tablet replica
I20250114 02:53:11.675063 425 raft_consensus.cc:2238] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 5 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.675451 425 raft_consensus.cc:2267] T 0ac89c2972b74708b89db61fb9f846ac P 23a08c9676c449e583b232736f562fad [term 5 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.676941 425 tablet_replica.cc:331] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad: stopping tablet replica
I20250114 02:53:11.677428 425 raft_consensus.cc:2238] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.677806 425 raft_consensus.cc:2267] T 287461283c6247658bdaca7184ac2067 P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.679627 425 tablet_replica.cc:331] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad: stopping tablet replica
I20250114 02:53:11.680061 425 raft_consensus.cc:2238] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.680465 425 raft_consensus.cc:2267] T eade2a70e5d54eacb0d30a272d470dfc P 23a08c9676c449e583b232736f562fad [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.681969 425 tablet_replica.cc:331] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad: stopping tablet replica
I20250114 02:53:11.682403 425 raft_consensus.cc:2238] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 3 LEADER]: Raft consensus shutting down.
I20250114 02:53:11.683159 425 raft_consensus.cc:2267] T 947325e5f22a41ed906165f55077bc67 P 23a08c9676c449e583b232736f562fad [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.684608 425 tablet_replica.cc:331] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad: stopping tablet replica
I20250114 02:53:11.684995 425 raft_consensus.cc:2238] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:53:11.685598 425 raft_consensus.cc:2267] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 23a08c9676c449e583b232736f562fad [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.687206 425 tablet_replica.cc:331] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad: stopping tablet replica
I20250114 02:53:11.687629 425 raft_consensus.cc:2238] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 4 LEADER]: Raft consensus shutting down.
I20250114 02:53:11.688369 425 raft_consensus.cc:2267] T 03c3a4647a594874889f28d2460a5566 P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.689870 425 tablet_replica.cc:331] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad: stopping tablet replica
I20250114 02:53:11.690307 425 raft_consensus.cc:2238] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.690698 425 raft_consensus.cc:2267] T 2b8a0de188ee481e8b79fd40d711382d P 23a08c9676c449e583b232736f562fad [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.720288 425 tablet_server.cc:195] TabletServer@127.0.106.66:0 shutdown complete.
I20250114 02:53:11.739617 425 tablet_server.cc:178] TabletServer@127.0.106.67:0 shutting down...
I20250114 02:53:11.763928 425 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 02:53:11.764564 425 tablet_replica.cc:331] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e: stopping tablet replica
I20250114 02:53:11.765359 425 raft_consensus.cc:2238] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 5 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.765900 425 raft_consensus.cc:2267] T 0ac89c2972b74708b89db61fb9f846ac P 0534eb8430324a1e83c9f26c3d46887e [term 5 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.767655 425 tablet_replica.cc:331] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e: stopping tablet replica
I20250114 02:53:11.768179 425 raft_consensus.cc:2238] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.768602 425 raft_consensus.cc:2267] T c2a601291dd147628db8399e31b46fe7 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.770169 425 tablet_replica.cc:331] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e: stopping tablet replica
I20250114 02:53:11.770699 425 raft_consensus.cc:2238] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 4 LEADER]: Raft consensus shutting down.
I20250114 02:53:11.771404 425 raft_consensus.cc:2267] T 287461283c6247658bdaca7184ac2067 P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.773051 425 tablet_replica.cc:331] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e: stopping tablet replica
I20250114 02:53:11.773532 425 raft_consensus.cc:2238] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.773898 425 raft_consensus.cc:2267] T 03c3a4647a594874889f28d2460a5566 P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.775444 425 tablet_replica.cc:331] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e: stopping tablet replica
I20250114 02:53:11.775938 425 raft_consensus.cc:2238] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.776342 425 raft_consensus.cc:2267] T 2b8a0de188ee481e8b79fd40d711382d P 0534eb8430324a1e83c9f26c3d46887e [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.777801 425 tablet_replica.cc:331] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e: stopping tablet replica
I20250114 02:53:11.778275 425 raft_consensus.cc:2238] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 2 LEADER]: Raft consensus shutting down.
I20250114 02:53:11.779026 425 raft_consensus.cc:2267] T 6acde705ff2a429985e29c3cb68c983d P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.780555 425 tablet_replica.cc:331] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e: stopping tablet replica
I20250114 02:53:11.780987 425 raft_consensus.cc:2238] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 2 LEADER]: Raft consensus shutting down.
I20250114 02:53:11.781639 425 raft_consensus.cc:2267] T eade2a70e5d54eacb0d30a272d470dfc P 0534eb8430324a1e83c9f26c3d46887e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.783186 425 tablet_replica.cc:331] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e: stopping tablet replica
I20250114 02:53:11.783589 425 raft_consensus.cc:2238] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.783973 425 raft_consensus.cc:2267] T 947325e5f22a41ed906165f55077bc67 P 0534eb8430324a1e83c9f26c3d46887e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.785398 425 tablet_replica.cc:331] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e: stopping tablet replica
I20250114 02:53:11.785806 425 raft_consensus.cc:2238] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:11.786155 425 raft_consensus.cc:2267] T 4fe43ac5a2984753aa6ad0f945a9d2a9 P 0534eb8430324a1e83c9f26c3d46887e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:11.811380 425 tablet_server.cc:195] TabletServer@127.0.106.67:0 shutdown complete.
I20250114 02:53:11.828333 425 master.cc:537] Master@127.0.106.126:45803 shutting down...
I20250114 02:53:12.018460 425 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:53:12.019119 425 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:12.019433 425 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7eb855ea5ab8482d8facc17c41aea2ac: stopping tablet replica
I20250114 02:53:12.037616 425 master.cc:559] Master@127.0.106.126:45803 shutdown complete.
0534eb8430324a1e83c9f26c3d46887e 1
23a08c9676c449e583b232736f562fad 4
ef76faab133b4cbf96d9e7610be21b62 4
0534eb8430324a1e83c9f26c3d46887e 3
23a08c9676c449e583b232736f562fad 3
ef76faab133b4cbf96d9e7610be21b62 3
0534eb8430324a1e83c9f26c3d46887e 1
23a08c9676c449e583b232736f562fad 8
ef76faab133b4cbf96d9e7610be21b62 0
0534eb8430324a1e83c9f26c3d46887e 3
23a08c9676c449e583b232736f562fad 3
ef76faab133b4cbf96d9e7610be21b62 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31046 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20250114 02:53:12.075554 425 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.106.126:38423
I20250114 02:53:12.076650 425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:53:12.082800 955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:12.083742 956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:12.084750 958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:12.085793 425 server_base.cc:1034] running on GCE node
I20250114 02:53:12.086815 425 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 02:53:12.087002 425 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 02:53:12.087122 425 hybrid_clock.cc:648] HybridClock initialized: now 1736823192087111 us; error 0 us; skew 500 ppm
I20250114 02:53:12.087548 425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:53:12.089809 425 webserver.cc:458] Webserver started at http://127.0.106.126:33085/ using document root <none> and password file <none>
I20250114 02:53:12.090267 425 fs_manager.cc:362] Metadata directory not provided
I20250114 02:53:12.090411 425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:53:12.090649 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:53:12.091686 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/master-0-root/instance:
uuid: "3d0f9c5915b6496bb155c7526d0128a9"
format_stamp: "Formatted at 2025-01-14 02:53:12 on dist-test-slave-k78q"
I20250114 02:53:12.095867 425 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20250114 02:53:12.098981 963 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:12.099712 425 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20250114 02:53:12.099965 425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/master-0-root
uuid: "3d0f9c5915b6496bb155c7526d0128a9"
format_stamp: "Formatted at 2025-01-14 02:53:12 on dist-test-slave-k78q"
I20250114 02:53:12.100231 425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:53:12.115599 425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:53:12.116829 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:53:12.152889 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.126:38423
I20250114 02:53:12.153033 1014 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.126:38423 every 8 connection(s)
I20250114 02:53:12.156563 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.166621 1015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9: Bootstrap starting.
I20250114 02:53:12.170895 1015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.174742 1015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9: No bootstrap required, opened a new log
I20250114 02:53:12.176622 1015 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0f9c5915b6496bb155c7526d0128a9" member_type: VOTER }
I20250114 02:53:12.177047 1015 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:12.177276 1015 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0f9c5915b6496bb155c7526d0128a9, State: Initialized, Role: FOLLOWER
I20250114 02:53:12.177752 1015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0f9c5915b6496bb155c7526d0128a9" member_type: VOTER }
I20250114 02:53:12.178175 1015 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 02:53:12.178385 1015 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 02:53:12.178674 1015 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:12.182150 1015 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0f9c5915b6496bb155c7526d0128a9" member_type: VOTER }
I20250114 02:53:12.182610 1015 leader_election.cc:304] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d0f9c5915b6496bb155c7526d0128a9; no voters:
I20250114 02:53:12.183713 1015 leader_election.cc:290] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 02:53:12.184084 1018 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:12.185304 1018 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [term 1 LEADER]: Becoming Leader. State: Replica: 3d0f9c5915b6496bb155c7526d0128a9, State: Running, Role: LEADER
I20250114 02:53:12.186049 1018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0f9c5915b6496bb155c7526d0128a9" member_type: VOTER }
I20250114 02:53:12.186445 1015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 02:53:12.190475 1019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d0f9c5915b6496bb155c7526d0128a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0f9c5915b6496bb155c7526d0128a9" member_type: VOTER } }
I20250114 02:53:12.190709 1020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0f9c5915b6496bb155c7526d0128a9. Latest consensus state: current_term: 1 leader_uuid: "3d0f9c5915b6496bb155c7526d0128a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0f9c5915b6496bb155c7526d0128a9" member_type: VOTER } }
I20250114 02:53:12.191118 1019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [sys.catalog]: This master's current role is: LEADER
I20250114 02:53:12.191207 1020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [sys.catalog]: This master's current role is: LEADER
I20250114 02:53:12.195966 1025 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 02:53:12.201553 1025 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 02:53:12.202448 425 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 02:53:12.209683 1025 catalog_manager.cc:1348] Generated new cluster ID: 0b700f3e5aa44a6195f1fc04ae467bd5
I20250114 02:53:12.209988 1025 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 02:53:12.224370 1025 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 02:53:12.225598 1025 catalog_manager.cc:1505] Loading token signing keys...
I20250114 02:53:12.239486 1025 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9: Generated new TSK 0
I20250114 02:53:12.240056 1025 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 02:53:12.269006 425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:53:12.275790 1036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:12.276726 1037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:12.277948 1039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:12.278862 425 server_base.cc:1034] running on GCE node
I20250114 02:53:12.279693 425 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 02:53:12.279882 425 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 02:53:12.280036 425 hybrid_clock.cc:648] HybridClock initialized: now 1736823192280016 us; error 0 us; skew 500 ppm
I20250114 02:53:12.280537 425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:53:12.282891 425 webserver.cc:458] Webserver started at http://127.0.106.65:41421/ using document root <none> and password file <none>
I20250114 02:53:12.283336 425 fs_manager.cc:362] Metadata directory not provided
I20250114 02:53:12.283507 425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:53:12.283756 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:53:12.284746 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-0-root/instance:
uuid: "fe3889e52a2c4bc2873b387533c90d34"
format_stamp: "Formatted at 2025-01-14 02:53:12 on dist-test-slave-k78q"
I20250114 02:53:12.289129 425 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250114 02:53:12.291954 1044 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:12.292649 425 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 02:53:12.292901 425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-0-root
uuid: "fe3889e52a2c4bc2873b387533c90d34"
format_stamp: "Formatted at 2025-01-14 02:53:12 on dist-test-slave-k78q"
I20250114 02:53:12.293176 425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:53:12.307703 425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:53:12.308862 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:53:12.310253 425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:53:12.312302 425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:53:12.312484 425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:12.312695 425 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:53:12.312829 425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:12.349218 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.65:42803
I20250114 02:53:12.349315 1106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.65:42803 every 8 connection(s)
I20250114 02:53:12.353591 425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:53:12.361778 1111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:12.362496 1112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:12.365098 1114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:12.365908 1107 heartbeater.cc:346] Connected to a master server at 127.0.106.126:38423
I20250114 02:53:12.366243 1107 heartbeater.cc:463] Registering TS with master...
I20250114 02:53:12.366904 1107 heartbeater.cc:510] Master 127.0.106.126:38423 requested a full tablet report, sending...
I20250114 02:53:12.367462 425 server_base.cc:1034] running on GCE node
I20250114 02:53:12.368772 425 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250114 02:53:12.368842 980 ts_manager.cc:194] Registered new tserver with Master: fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803)
W20250114 02:53:12.369056 425 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 02:53:12.369273 425 hybrid_clock.cc:648] HybridClock initialized: now 1736823192369254 us; error 0 us; skew 500 ppm
I20250114 02:53:12.369915 425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:53:12.370752 980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54966
I20250114 02:53:12.372557 425 webserver.cc:458] Webserver started at http://127.0.106.66:33403/ using document root <none> and password file <none>
I20250114 02:53:12.373055 425 fs_manager.cc:362] Metadata directory not provided
I20250114 02:53:12.373243 425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:53:12.373471 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:53:12.374516 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-1-root/instance:
uuid: "678704d2559e41eb9d8b76af869bbefc"
format_stamp: "Formatted at 2025-01-14 02:53:12 on dist-test-slave-k78q"
I20250114 02:53:12.378587 425 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250114 02:53:12.381528 1119 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:12.382193 425 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 02:53:12.382443 425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-1-root
uuid: "678704d2559e41eb9d8b76af869bbefc"
format_stamp: "Formatted at 2025-01-14 02:53:12 on dist-test-slave-k78q"
I20250114 02:53:12.382745 425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:53:12.401499 425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:53:12.402683 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:53:12.404017 425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:53:12.406092 425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:53:12.406280 425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:12.406500 425 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:53:12.406672 425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250114 02:53:12.443375 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.66:34721
I20250114 02:53:12.443467 1181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.66:34721 every 8 connection(s)
I20250114 02:53:12.447897 425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:53:12.455464 1185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:12.456490 1186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:12.457585 1188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:12.458601 425 server_base.cc:1034] running on GCE node
I20250114 02:53:12.459398 1182 heartbeater.cc:346] Connected to a master server at 127.0.106.126:38423
I20250114 02:53:12.459561 425 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250114 02:53:12.459712 1182 heartbeater.cc:463] Registering TS with master...
W20250114 02:53:12.459823 425 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 02:53:12.460040 425 hybrid_clock.cc:648] HybridClock initialized: now 1736823192460024 us; error 0 us; skew 500 ppm
I20250114 02:53:12.460471 1182 heartbeater.cc:510] Master 127.0.106.126:38423 requested a full tablet report, sending...
I20250114 02:53:12.460536 425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:53:12.462630 980 ts_manager.cc:194] Registered new tserver with Master: 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:12.463488 425 webserver.cc:458] Webserver started at http://127.0.106.67:42315/ using document root <none> and password file <none>
I20250114 02:53:12.464100 425 fs_manager.cc:362] Metadata directory not provided
I20250114 02:53:12.464320 425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:53:12.464380 980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54980
I20250114 02:53:12.464668 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:53:12.466059 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-2-root/instance:
uuid: "dba85f60dcc94b6797318aef42473236"
format_stamp: "Formatted at 2025-01-14 02:53:12 on dist-test-slave-k78q"
I20250114 02:53:12.470193 425 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250114 02:53:12.473323 1193 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:12.474028 425 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250114 02:53:12.474296 425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-2-root
uuid: "dba85f60dcc94b6797318aef42473236"
format_stamp: "Formatted at 2025-01-14 02:53:12 on dist-test-slave-k78q"
I20250114 02:53:12.474555 425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1736823160872850-425-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:53:12.483337 425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:53:12.484404 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:53:12.485745 425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:53:12.487761 425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:53:12.487962 425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:12.488180 425 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:53:12.488325 425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:12.525704 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.67:34357
I20250114 02:53:12.525791 1255 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.67:34357 every 8 connection(s)
I20250114 02:53:12.538378 1256 heartbeater.cc:346] Connected to a master server at 127.0.106.126:38423
I20250114 02:53:12.538733 1256 heartbeater.cc:463] Registering TS with master...
I20250114 02:53:12.539419 1256 heartbeater.cc:510] Master 127.0.106.126:38423 requested a full tablet report, sending...
I20250114 02:53:12.541057 980 ts_manager.cc:194] Registered new tserver with Master: dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:12.541518 425 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012882348s
I20250114 02:53:12.542264 980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54982
I20250114 02:53:12.543815 425 test_util.cc:274] Using random seed: -1452416459
I20250114 02:53:12.564713 980 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54992:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:53:12.567478 980 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:53:12.640686 1147 tablet_service.cc:1467] Processing CreateTablet for tablet 3144d42016f44d539f4c03185a17cd64 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250114 02:53:12.642144 1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3144d42016f44d539f4c03185a17cd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.642443 1070 tablet_service.cc:1467] Processing CreateTablet for tablet 559ae9b489104111a990c10573de9259 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250114 02:53:12.643795 1070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 559ae9b489104111a990c10573de9259. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.645188 1072 tablet_service.cc:1467] Processing CreateTablet for tablet 3144d42016f44d539f4c03185a17cd64 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250114 02:53:12.647734 1146 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3949440ca34a658306d0065161a875 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250114 02:53:12.649106 1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3949440ca34a658306d0065161a875. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.648905 1071 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3949440ca34a658306d0065161a875 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250114 02:53:12.647729 1144 tablet_service.cc:1467] Processing CreateTablet for tablet 3ccf3b62f4d44f9d85522e68a32a800f (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250114 02:53:12.651144 1143 tablet_service.cc:1467] Processing CreateTablet for tablet bdc1d29533f046f9997fc1dd89c84722 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250114 02:53:12.651955 1144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ccf3b62f4d44f9d85522e68a32a800f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.652586 1143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdc1d29533f046f9997fc1dd89c84722. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.653254 1142 tablet_service.cc:1467] Processing CreateTablet for tablet db1518c696914d80b37027e1b8d4fda0 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250114 02:53:12.654520 1142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db1518c696914d80b37027e1b8d4fda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.643363 1145 tablet_service.cc:1467] Processing CreateTablet for tablet 559ae9b489104111a990c10573de9259 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250114 02:53:12.663338 1145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 559ae9b489104111a990c10573de9259. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.664347 1218 tablet_service.cc:1467] Processing CreateTablet for tablet 3ccf3b62f4d44f9d85522e68a32a800f (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250114 02:53:12.665691 1218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ccf3b62f4d44f9d85522e68a32a800f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.658685 1067 tablet_service.cc:1467] Processing CreateTablet for tablet db1518c696914d80b37027e1b8d4fda0 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250114 02:53:12.660293 1068 tablet_service.cc:1467] Processing CreateTablet for tablet bdc1d29533f046f9997fc1dd89c84722 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250114 02:53:12.661978 1069 tablet_service.cc:1467] Processing CreateTablet for tablet 3ccf3b62f4d44f9d85522e68a32a800f (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250114 02:53:12.646414 1072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3144d42016f44d539f4c03185a17cd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.668995 1217 tablet_service.cc:1467] Processing CreateTablet for tablet bdc1d29533f046f9997fc1dd89c84722 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250114 02:53:12.677318 1219 tablet_service.cc:1467] Processing CreateTablet for tablet 559ae9b489104111a990c10573de9259 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250114 02:53:12.677309 1216 tablet_service.cc:1467] Processing CreateTablet for tablet db1518c696914d80b37027e1b8d4fda0 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250114 02:53:12.679842 1066 tablet_service.cc:1467] Processing CreateTablet for tablet bfb6d62d39fa47059bcf3d88f235c999 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250114 02:53:12.682900 1219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 559ae9b489104111a990c10573de9259. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.683086 1141 tablet_service.cc:1467] Processing CreateTablet for tablet bfb6d62d39fa47059bcf3d88f235c999 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250114 02:53:12.683717 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3949440ca34a658306d0065161a875 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250114 02:53:12.684394 1141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfb6d62d39fa47059bcf3d88f235c999. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.685096 1220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3949440ca34a658306d0065161a875. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.686741 1065 tablet_service.cc:1467] Processing CreateTablet for tablet 8452bf971d30409e81d5d997e366ed89 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250114 02:53:12.688066 1065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8452bf971d30409e81d5d997e366ed89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.687758 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 3144d42016f44d539f4c03185a17cd64 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250114 02:53:12.689067 1068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdc1d29533f046f9997fc1dd89c84722. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.689210 1221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3144d42016f44d539f4c03185a17cd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.690663 1066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfb6d62d39fa47059bcf3d88f235c999. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.690997 1216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db1518c696914d80b37027e1b8d4fda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.695609 1067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db1518c696914d80b37027e1b8d4fda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.695771 1217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdc1d29533f046f9997fc1dd89c84722. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.699347 1069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ccf3b62f4d44f9d85522e68a32a800f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.709486 1215 tablet_service.cc:1467] Processing CreateTablet for tablet bfb6d62d39fa47059bcf3d88f235c999 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250114 02:53:12.710891 1215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfb6d62d39fa47059bcf3d88f235c999. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.706864 1071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3949440ca34a658306d0065161a875. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.726292 1272 tablet_bootstrap.cc:492] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc: Bootstrap starting.
I20250114 02:53:12.744589 1271 tablet_bootstrap.cc:492] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34: Bootstrap starting.
I20250114 02:53:12.756279 1140 tablet_service.cc:1467] Processing CreateTablet for tablet bebf4d4a188940c685b6726024522268 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250114 02:53:12.757571 1140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bebf4d4a188940c685b6726024522268. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.752825 1145 tablet_service.cc:1467] Processing CreateTablet for tablet 8452bf971d30409e81d5d997e366ed89 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250114 02:53:12.765275 1145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8452bf971d30409e81d5d997e366ed89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.771276 1271 tablet_bootstrap.cc:654] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.772480 1072 tablet_service.cc:1467] Processing CreateTablet for tablet bebf4d4a188940c685b6726024522268 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250114 02:53:12.777415 1072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bebf4d4a188940c685b6726024522268. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.773943 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 8452bf971d30409e81d5d997e366ed89 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250114 02:53:12.779312 1220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8452bf971d30409e81d5d997e366ed89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.799139 1221 tablet_service.cc:1467] Processing CreateTablet for tablet bebf4d4a188940c685b6726024522268 (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250114 02:53:12.800383 1221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bebf4d4a188940c685b6726024522268. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.798666 1272 tablet_bootstrap.cc:654] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.806500 1273 tablet_bootstrap.cc:492] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236: Bootstrap starting.
I20250114 02:53:12.818528 1271 tablet_bootstrap.cc:492] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34: No bootstrap required, opened a new log
I20250114 02:53:12.821590 1273 tablet_bootstrap.cc:654] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.825357 1271 ts_tablet_manager.cc:1397] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34: Time spent bootstrapping tablet: real 0.082s user 0.011s sys 0.002s
I20250114 02:53:12.823585 1219 tablet_service.cc:1467] Processing CreateTablet for tablet 7415008b1fac4680894ed783f47e65fd (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250114 02:53:12.823820 1141 tablet_service.cc:1467] Processing CreateTablet for tablet 7415008b1fac4680894ed783f47e65fd (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250114 02:53:12.827752 1141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7415008b1fac4680894ed783f47e65fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.829300 1271 raft_consensus.cc:357] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.830041 1271 raft_consensus.cc:383] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:12.830328 1271 raft_consensus.cc:738] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Initialized, Role: FOLLOWER
I20250114 02:53:12.831667 1066 tablet_service.cc:1467] Processing CreateTablet for tablet 7415008b1fac4680894ed783f47e65fd (DEFAULT_TABLE table=test-workload [id=80aa4f85b6f34c0ba1edd9c602cade0e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250114 02:53:12.832813 1271 consensus_queue.cc:260] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.835001 1219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7415008b1fac4680894ed783f47e65fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.835150 1272 tablet_bootstrap.cc:492] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc: No bootstrap required, opened a new log
I20250114 02:53:12.835732 1272 ts_tablet_manager.cc:1397] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc: Time spent bootstrapping tablet: real 0.110s user 0.002s sys 0.012s
I20250114 02:53:12.839464 1272 raft_consensus.cc:357] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.840163 1272 raft_consensus.cc:383] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:12.840459 1272 raft_consensus.cc:738] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Initialized, Role: FOLLOWER
I20250114 02:53:12.840618 1066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7415008b1fac4680894ed783f47e65fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:12.841259 1272 consensus_queue.cc:260] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.858768 1272 ts_tablet_manager.cc:1428] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc: Time spent starting tablet: real 0.023s user 0.006s sys 0.005s
I20250114 02:53:12.859023 1271 ts_tablet_manager.cc:1428] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34: Time spent starting tablet: real 0.033s user 0.012s sys 0.007s
I20250114 02:53:12.859642 1107 heartbeater.cc:502] Master 127.0.106.126:38423 was elected leader, sending a full tablet report...
I20250114 02:53:12.859781 1273 tablet_bootstrap.cc:492] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236: No bootstrap required, opened a new log
I20250114 02:53:12.860291 1272 tablet_bootstrap.cc:492] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc: Bootstrap starting.
I20250114 02:53:12.859642 1182 heartbeater.cc:502] Master 127.0.106.126:38423 was elected leader, sending a full tablet report...
I20250114 02:53:12.866734 1272 tablet_bootstrap.cc:654] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.860296 1273 ts_tablet_manager.cc:1397] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236: Time spent bootstrapping tablet: real 0.054s user 0.020s sys 0.015s
I20250114 02:53:12.871201 1271 tablet_bootstrap.cc:492] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34: Bootstrap starting.
I20250114 02:53:12.872025 1273 raft_consensus.cc:357] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.872749 1273 raft_consensus.cc:383] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:12.873029 1273 raft_consensus.cc:738] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Initialized, Role: FOLLOWER
I20250114 02:53:12.873747 1273 consensus_queue.cc:260] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.877074 1256 heartbeater.cc:502] Master 127.0.106.126:38423 was elected leader, sending a full tablet report...
I20250114 02:53:12.877694 1273 ts_tablet_manager.cc:1428] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250114 02:53:12.878554 1273 tablet_bootstrap.cc:492] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236: Bootstrap starting.
I20250114 02:53:12.879334 1271 tablet_bootstrap.cc:654] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.882930 1272 tablet_bootstrap.cc:492] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc: No bootstrap required, opened a new log
I20250114 02:53:12.883388 1272 ts_tablet_manager.cc:1397] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.004s
I20250114 02:53:12.886132 1272 raft_consensus.cc:357] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.886914 1272 raft_consensus.cc:383] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:12.887217 1272 raft_consensus.cc:738] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Initialized, Role: FOLLOWER
I20250114 02:53:12.887882 1272 consensus_queue.cc:260] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.889693 1272 ts_tablet_manager.cc:1428] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:53:12.890368 1272 tablet_bootstrap.cc:492] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc: Bootstrap starting.
I20250114 02:53:12.893575 1273 tablet_bootstrap.cc:654] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.901592 1273 tablet_bootstrap.cc:492] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236: No bootstrap required, opened a new log
I20250114 02:53:12.902124 1273 ts_tablet_manager.cc:1397] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.000s
I20250114 02:53:12.902621 1272 tablet_bootstrap.cc:654] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.904820 1273 raft_consensus.cc:357] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.905709 1273 raft_consensus.cc:383] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:12.906092 1273 raft_consensus.cc:738] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Initialized, Role: FOLLOWER
I20250114 02:53:12.907028 1273 consensus_queue.cc:260] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.910243 1273 ts_tablet_manager.cc:1428] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250114 02:53:12.911339 1273 tablet_bootstrap.cc:492] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236: Bootstrap starting.
I20250114 02:53:12.919507 1273 tablet_bootstrap.cc:654] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.927209 1272 tablet_bootstrap.cc:492] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc: No bootstrap required, opened a new log
I20250114 02:53:12.927273 1273 tablet_bootstrap.cc:492] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236: No bootstrap required, opened a new log
I20250114 02:53:12.927269 1271 tablet_bootstrap.cc:492] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34: No bootstrap required, opened a new log
I20250114 02:53:12.927662 1272 ts_tablet_manager.cc:1397] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.014s
I20250114 02:53:12.927731 1273 ts_tablet_manager.cc:1397] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20250114 02:53:12.927973 1271 ts_tablet_manager.cc:1397] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34: Time spent bootstrapping tablet: real 0.057s user 0.019s sys 0.033s
I20250114 02:53:12.930372 1271 raft_consensus.cc:357] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.930354 1273 raft_consensus.cc:357] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.931102 1271 raft_consensus.cc:383] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:12.930773 1272 raft_consensus.cc:357] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.931260 1273 raft_consensus.cc:383] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:12.931411 1271 raft_consensus.cc:738] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Initialized, Role: FOLLOWER
I20250114 02:53:12.931509 1272 raft_consensus.cc:383] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:12.931588 1273 raft_consensus.cc:738] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Initialized, Role: FOLLOWER
I20250114 02:53:12.931913 1272 raft_consensus.cc:738] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Initialized, Role: FOLLOWER
I20250114 02:53:12.932125 1271 consensus_queue.cc:260] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.932359 1273 consensus_queue.cc:260] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.932582 1272 consensus_queue.cc:260] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.934202 1271 ts_tablet_manager.cc:1428] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250114 02:53:12.934829 1273 ts_tablet_manager.cc:1428] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250114 02:53:12.935115 1271 tablet_bootstrap.cc:492] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34: Bootstrap starting.
I20250114 02:53:12.935851 1273 tablet_bootstrap.cc:492] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236: Bootstrap starting.
I20250114 02:53:12.937516 1272 ts_tablet_manager.cc:1428] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250114 02:53:12.938464 1272 tablet_bootstrap.cc:492] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc: Bootstrap starting.
I20250114 02:53:12.942773 1271 tablet_bootstrap.cc:654] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.943176 1273 tablet_bootstrap.cc:654] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.944689 1272 tablet_bootstrap.cc:654] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.969419 1271 tablet_bootstrap.cc:492] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34: No bootstrap required, opened a new log
I20250114 02:53:12.969890 1271 ts_tablet_manager.cc:1397] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.019s
I20250114 02:53:12.970763 1273 tablet_bootstrap.cc:492] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236: No bootstrap required, opened a new log
I20250114 02:53:12.971132 1273 ts_tablet_manager.cc:1397] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.019s
I20250114 02:53:12.972641 1271 raft_consensus.cc:357] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.973423 1271 raft_consensus.cc:383] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:12.973699 1271 raft_consensus.cc:738] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Initialized, Role: FOLLOWER
I20250114 02:53:12.973302 1273 raft_consensus.cc:357] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.974165 1273 raft_consensus.cc:383] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:12.974283 1271 consensus_queue.cc:260] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.974680 1272 tablet_bootstrap.cc:492] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc: No bootstrap required, opened a new log
I20250114 02:53:12.974500 1273 raft_consensus.cc:738] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Initialized, Role: FOLLOWER
I20250114 02:53:12.975178 1272 ts_tablet_manager.cc:1397] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.018s
I20250114 02:53:12.975389 1273 consensus_queue.cc:260] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.976605 1271 ts_tablet_manager.cc:1428] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250114 02:53:12.977494 1271 tablet_bootstrap.cc:492] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34: Bootstrap starting.
I20250114 02:53:12.977831 1273 ts_tablet_manager.cc:1428] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 02:53:12.977965 1272 raft_consensus.cc:357] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.978730 1272 raft_consensus.cc:383] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:12.978809 1273 tablet_bootstrap.cc:492] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236: Bootstrap starting.
I20250114 02:53:12.979110 1272 raft_consensus.cc:738] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Initialized, Role: FOLLOWER
I20250114 02:53:12.979908 1272 consensus_queue.cc:260] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:12.983985 1271 tablet_bootstrap.cc:654] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.985057 1273 tablet_bootstrap.cc:654] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:12.989073 1272 ts_tablet_manager.cc:1428] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20250114 02:53:12.995939 1272 tablet_bootstrap.cc:492] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc: Bootstrap starting.
I20250114 02:53:13.002403 1272 tablet_bootstrap.cc:654] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.006831 1271 tablet_bootstrap.cc:492] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34: No bootstrap required, opened a new log
I20250114 02:53:13.007257 1271 ts_tablet_manager.cc:1397] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.020s
I20250114 02:53:13.007532 1273 tablet_bootstrap.cc:492] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236: No bootstrap required, opened a new log
I20250114 02:53:13.007936 1273 ts_tablet_manager.cc:1397] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.010s
I20250114 02:53:13.008117 1272 tablet_bootstrap.cc:492] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc: No bootstrap required, opened a new log
I20250114 02:53:13.008481 1272 ts_tablet_manager.cc:1397] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20250114 02:53:13.009688 1271 raft_consensus.cc:357] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.010411 1271 raft_consensus.cc:383] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.010871 1271 raft_consensus.cc:738] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.010494 1272 raft_consensus.cc:357] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.010423 1273 raft_consensus.cc:357] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.011354 1272 raft_consensus.cc:383] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.011447 1273 raft_consensus.cc:383] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.011768 1272 raft_consensus.cc:738] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.011804 1273 raft_consensus.cc:738] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.011672 1271 consensus_queue.cc:260] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.012555 1272 consensus_queue.cc:260] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.012672 1273 consensus_queue.cc:260] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.014073 1271 ts_tablet_manager.cc:1428] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250114 02:53:13.014927 1271 tablet_bootstrap.cc:492] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34: Bootstrap starting.
I20250114 02:53:13.016755 1272 ts_tablet_manager.cc:1428] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250114 02:53:13.017522 1273 ts_tablet_manager.cc:1428] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250114 02:53:13.017611 1272 tablet_bootstrap.cc:492] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc: Bootstrap starting.
I20250114 02:53:13.018489 1273 tablet_bootstrap.cc:492] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236: Bootstrap starting.
I20250114 02:53:13.020959 1271 tablet_bootstrap.cc:654] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.023954 1277 raft_consensus.cc:491] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.024216 1272 tablet_bootstrap.cc:654] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.024437 1277 raft_consensus.cc:513] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.025665 1273 tablet_bootstrap.cc:654] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.034698 1277 leader_election.cc:290] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.037433 1273 tablet_bootstrap.cc:492] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236: No bootstrap required, opened a new log
I20250114 02:53:13.037967 1273 ts_tablet_manager.cc:1397] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20250114 02:53:13.040874 1273 raft_consensus.cc:357] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.041555 1273 raft_consensus.cc:383] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.041846 1273 raft_consensus.cc:738] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.043576 1280 raft_consensus.cc:491] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.044137 1280 raft_consensus.cc:513] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.046941 1272 tablet_bootstrap.cc:492] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc: No bootstrap required, opened a new log
I20250114 02:53:13.047435 1272 ts_tablet_manager.cc:1397] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.012s
I20250114 02:53:13.047850 1278 raft_consensus.cc:491] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.048301 1278 raft_consensus.cc:513] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.050122 1272 raft_consensus.cc:357] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.050869 1272 raft_consensus.cc:383] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.051134 1272 raft_consensus.cc:738] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.051899 1278 leader_election.cc:290] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.051801 1272 consensus_queue.cc:260] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.055217 1272 ts_tablet_manager.cc:1428] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250114 02:53:13.056209 1272 tablet_bootstrap.cc:492] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc: Bootstrap starting.
I20250114 02:53:13.057682 1280 leader_election.cc:290] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:13.063596 1272 tablet_bootstrap.cc:654] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.072227 1277 raft_consensus.cc:491] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.072821 1277 raft_consensus.cc:513] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.042874 1273 consensus_queue.cc:260] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.075491 1277 leader_election.cc:290] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.078295 1273 ts_tablet_manager.cc:1428] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236: Time spent starting tablet: real 0.040s user 0.008s sys 0.000s
I20250114 02:53:13.079322 1273 tablet_bootstrap.cc:492] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236: Bootstrap starting.
I20250114 02:53:13.096144 1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccf3b62f4d44f9d85522e68a32a800f" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc" is_pre_election: true
I20250114 02:53:13.097201 1157 raft_consensus.cc:2463] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 0.
I20250114 02:53:13.098773 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc" is_pre_election: true
I20250114 02:53:13.099633 1156 raft_consensus.cc:2463] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 0.
I20250114 02:53:13.099997 1045 leader_election.cc:304] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:13.100983 1277 raft_consensus.cc:2798] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:13.101346 1277 raft_consensus.cc:491] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:13.101668 1277 raft_consensus.cc:3054] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.102878 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccf3b62f4d44f9d85522e68a32a800f" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236" is_pre_election: true
I20250114 02:53:13.103592 1231 raft_consensus.cc:2463] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 0.
I20250114 02:53:13.104995 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236" is_pre_election: true
W20250114 02:53:13.106978 1047 leader_election.cc:343] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.108600 1277 raft_consensus.cc:513] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.110814 1277 leader_election.cc:290] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 election: Requested vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.112331 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccf3b62f4d44f9d85522e68a32a800f" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:13.113037 1231 raft_consensus.cc:3054] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.124519 1278 raft_consensus.cc:491] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.125011 1278 raft_consensus.cc:513] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.126904 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccf3b62f4d44f9d85522e68a32a800f" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:13.127492 1156 raft_consensus.cc:3054] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.127712 1045 leader_election.cc:304] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, fe3889e52a2c4bc2873b387533c90d34; no voters: dba85f60dcc94b6797318aef42473236
I20250114 02:53:13.128809 1277 raft_consensus.cc:2798] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:13.129185 1277 raft_consensus.cc:491] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:13.129503 1277 raft_consensus.cc:3054] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.127063 1278 leader_election.cc:290] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.131510 1273 tablet_bootstrap.cc:654] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.122154 1231 raft_consensus.cc:2463] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 1.
I20250114 02:53:13.134784 1047 leader_election.cc:304] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:13.135960 1277 raft_consensus.cc:513] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.136961 1156 raft_consensus.cc:2463] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 1.
I20250114 02:53:13.140331 1271 tablet_bootstrap.cc:492] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34: No bootstrap required, opened a new log
I20250114 02:53:13.140873 1271 ts_tablet_manager.cc:1397] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34: Time spent bootstrapping tablet: real 0.126s user 0.016s sys 0.025s
I20250114 02:53:13.143747 1271 raft_consensus.cc:357] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.144472 1271 raft_consensus.cc:383] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.144809 1271 raft_consensus.cc:738] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.144956 1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3949440ca34a658306d0065161a875" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc" is_pre_election: true
I20250114 02:53:13.145555 1271 consensus_queue.cc:260] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.147440 1277 leader_election.cc:290] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 election: Requested vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.147622 1305 raft_consensus.cc:2798] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:13.148065 1305 raft_consensus.cc:695] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Becoming Leader. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:13.148802 1305 consensus_queue.cc:237] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.152083 1277 raft_consensus.cc:491] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.152264 1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:13.153064 1157 raft_consensus.cc:3054] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.152635 1277 raft_consensus.cc:513] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.156525 1277 leader_election.cc:290] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.156806 1271 ts_tablet_manager.cc:1428] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34: Time spent starting tablet: real 0.016s user 0.008s sys 0.002s
I20250114 02:53:13.157899 1271 tablet_bootstrap.cc:492] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34: Bootstrap starting.
I20250114 02:53:13.158078 1082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3949440ca34a658306d0065161a875" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34" is_pre_election: true
W20250114 02:53:13.159778 1195 leader_election.cc:343] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.161023 1230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ae9b489104111a990c10573de9259" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236" is_pre_election: true
I20250114 02:53:13.161661 1230 raft_consensus.cc:2463] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 0.
I20250114 02:53:13.163028 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:13.164340 1271 tablet_bootstrap.cc:654] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.164438 1047 leader_election.cc:304] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:13.166864 1277 raft_consensus.cc:2798] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:13.167218 1277 raft_consensus.cc:491] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:13.167541 1277 raft_consensus.cc:3054] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.167817 1278 raft_consensus.cc:491] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:53:13.167912 1194 leader_election.cc:343] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.168323 1278 raft_consensus.cc:513] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.168496 1194 leader_election.cc:304] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dba85f60dcc94b6797318aef42473236; no voters: 678704d2559e41eb9d8b76af869bbefc, fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:13.169786 1280 raft_consensus.cc:2743] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:13.170246 1278 leader_election.cc:290] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.171027 1280 raft_consensus.cc:491] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.171516 1280 raft_consensus.cc:513] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.173751 1280 leader_election.cc:290] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
W20250114 02:53:13.173861 1047 leader_election.cc:343] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.177448 1155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc" is_pre_election: true
I20250114 02:53:13.178599 1155 raft_consensus.cc:2463] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 0.
I20250114 02:53:13.180650 1271 tablet_bootstrap.cc:492] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34: No bootstrap required, opened a new log
I20250114 02:53:13.181077 1271 ts_tablet_manager.cc:1397] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.000s
I20250114 02:53:13.181231 1194 leader_election.cc:304] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters:
I20250114 02:53:13.182355 1280 raft_consensus.cc:2798] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:13.182816 1280 raft_consensus.cc:491] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:13.183233 1280 raft_consensus.cc:3054] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.183704 1271 raft_consensus.cc:357] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.184398 1271 raft_consensus.cc:383] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.184681 1271 raft_consensus.cc:738] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.186476 1271 consensus_queue.cc:260] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.189155 1271 ts_tablet_manager.cc:1428] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250114 02:53:13.190058 1271 tablet_bootstrap.cc:492] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34: Bootstrap starting.
W20250114 02:53:13.198944 1028 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:53:13.193087 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ae9b489104111a990c10573de9259" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc" is_pre_election: true
I20250114 02:53:13.205801 1156 raft_consensus.cc:2463] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 0.
I20250114 02:53:13.210026 1082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34" is_pre_election: true
I20250114 02:53:13.210152 1277 raft_consensus.cc:513] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.210763 1082 raft_consensus.cc:2463] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 0.
I20250114 02:53:13.211097 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ae9b489104111a990c10573de9259" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34" is_pre_election: true
I20250114 02:53:13.211047 1280 raft_consensus.cc:513] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.211951 1081 raft_consensus.cc:2388] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 678704d2559e41eb9d8b76af869bbefc in current term 1: Already voted for candidate fe3889e52a2c4bc2873b387533c90d34 in this term.
I20250114 02:53:13.212034 1277 leader_election.cc:290] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 election: Requested vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.212949 1280 leader_election.cc:290] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:13.213006 1080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf4d4a188940c685b6726024522268" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34" is_pre_election: true
I20250114 02:53:13.213127 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ae9b489104111a990c10573de9259" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:13.213552 1157 raft_consensus.cc:2463] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 1.
I20250114 02:53:13.215780 1231 raft_consensus.cc:3054] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.216238 1230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ae9b489104111a990c10573de9259" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236" is_pre_election: true
I20250114 02:53:13.218185 1273 tablet_bootstrap.cc:492] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236: No bootstrap required, opened a new log
I20250114 02:53:13.218201 1272 tablet_bootstrap.cc:492] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc: No bootstrap required, opened a new log
I20250114 02:53:13.218786 1273 ts_tablet_manager.cc:1397] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236: Time spent bootstrapping tablet: real 0.140s user 0.011s sys 0.000s
I20250114 02:53:13.219266 1272 ts_tablet_manager.cc:1397] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc: Time spent bootstrapping tablet: real 0.163s user 0.063s sys 0.082s
I20250114 02:53:13.221129 1045 leader_election.cc:304] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, fe3889e52a2c4bc2873b387533c90d34; no voters: dba85f60dcc94b6797318aef42473236
I20250114 02:53:13.221590 1273 raft_consensus.cc:357] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.222322 1273 raft_consensus.cc:383] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.222898 1121 leader_election.cc:304] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc; no voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:13.225446 1305 raft_consensus.cc:2798] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:13.225651 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
W20250114 02:53:13.226200 1121 leader_election.cc:343] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.227073 1273 raft_consensus.cc:738] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.223883 1229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236" is_pre_election: true
I20250114 02:53:13.228294 1271 tablet_bootstrap.cc:654] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.227834 1273 consensus_queue.cc:260] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
W20250114 02:53:13.228665 1122 leader_election.cc:343] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.224637 1230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf4d4a188940c685b6726024522268" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236" is_pre_election: true
I20250114 02:53:13.224689 1080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34" is_pre_election: true
I20250114 02:53:13.229626 1231 raft_consensus.cc:2463] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 1.
I20250114 02:53:13.230252 1273 ts_tablet_manager.cc:1428] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
W20250114 02:53:13.230685 1122 leader_election.cc:343] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.231119 1047 leader_election.cc:304] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:13.226243 1081 raft_consensus.cc:3054] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.226238 1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ae9b489104111a990c10573de9259" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:13.226717 1278 raft_consensus.cc:3054] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.222357 1272 raft_consensus.cc:357] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.231108 1122 leader_election.cc:304] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc; no voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:13.224176 978 catalog_manager.cc:5526] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65). New cstate: current_term: 1 leader_uuid: "fe3889e52a2c4bc2873b387533c90d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:13.262305 1273 tablet_bootstrap.cc:492] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236: Bootstrap starting.
I20250114 02:53:13.227049 1305 raft_consensus.cc:695] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Becoming Leader. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:13.265632 1272 raft_consensus.cc:383] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.259413 1277 raft_consensus.cc:491] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.233124 1307 raft_consensus.cc:2798] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:13.266898 1272 raft_consensus.cc:738] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.267289 1307 raft_consensus.cc:695] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Becoming Leader. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:13.267642 1272 consensus_queue.cc:260] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.266181 1277 raft_consensus.cc:513] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.268072 1307 consensus_queue.cc:237] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.250015 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:13.270210 1277 leader_election.cc:290] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.269760 1156 raft_consensus.cc:3054] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.271591 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3144d42016f44d539f4c03185a17cd64" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236" is_pre_election: true
I20250114 02:53:13.273865 1231 raft_consensus.cc:2463] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 0.
W20250114 02:53:13.240154 1121 leader_election.cc:343] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.274993 1272 ts_tablet_manager.cc:1428] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc: Time spent starting tablet: real 0.055s user 0.019s sys 0.016s
I20250114 02:53:13.274847 1121 leader_election.cc:304] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc; no voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:13.277526 1047 leader_election.cc:304] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:13.264911 1278 raft_consensus.cc:2743] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:13.271799 1155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3144d42016f44d539f4c03185a17cd64" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc" is_pre_election: true
I20250114 02:53:13.256875 1081 raft_consensus.cc:2463] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 1.
I20250114 02:53:13.286127 1272 tablet_bootstrap.cc:492] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc: Bootstrap starting.
W20250114 02:53:13.286636 1121 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:42803, user_credentials={real_user=slave}} blocked reactor thread for 46566.4us
I20250114 02:53:13.282956 1277 raft_consensus.cc:2798] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:13.291203 1195 leader_election.cc:304] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34; no voters:
W20250114 02:53:13.285049 1045 leader_election.cc:343] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.292652 1280 raft_consensus.cc:2798] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:13.289928 1277 raft_consensus.cc:491] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 02:53:13.295213 1122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:34357, user_credentials={real_user=slave}} blocked reactor thread for 64558.7us
I20250114 02:53:13.296170 1277 raft_consensus.cc:3054] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.281698 1305 consensus_queue.cc:237] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.314014 1271 tablet_bootstrap.cc:492] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34: No bootstrap required, opened a new log
I20250114 02:53:13.314435 1271 ts_tablet_manager.cc:1397] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34: Time spent bootstrapping tablet: real 0.125s user 0.026s sys 0.033s
I20250114 02:53:13.314736 1311 raft_consensus.cc:2743] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:13.314879 1157 raft_consensus.cc:2463] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 1.
I20250114 02:53:13.315981 1273 tablet_bootstrap.cc:654] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.314877 1272 tablet_bootstrap.cc:654] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.317225 1271 raft_consensus.cc:357] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.317962 1271 raft_consensus.cc:383] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.318245 1271 raft_consensus.cc:738] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.310905 1277 raft_consensus.cc:513] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.320883 1313 raft_consensus.cc:2743] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:13.321380 1156 raft_consensus.cc:2463] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 1.
I20250114 02:53:13.320374 1280 raft_consensus.cc:695] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Becoming Leader. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:13.322479 1271 consensus_queue.cc:260] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.323783 1280 consensus_queue.cc:237] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.325526 1271 ts_tablet_manager.cc:1428] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250114 02:53:13.326761 1271 tablet_bootstrap.cc:492] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34: Bootstrap starting.
I20250114 02:53:13.335044 1273 tablet_bootstrap.cc:492] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236: No bootstrap required, opened a new log
I20250114 02:53:13.335383 1273 ts_tablet_manager.cc:1397] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236: Time spent bootstrapping tablet: real 0.073s user 0.011s sys 0.001s
I20250114 02:53:13.336971 1273 raft_consensus.cc:357] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.337424 1273 raft_consensus.cc:383] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.337644 1273 raft_consensus.cc:738] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.338481 1316 raft_consensus.cc:491] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.338338 1273 consensus_queue.cc:260] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.339514 1277 leader_election.cc:290] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 election: Requested vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.340313 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3144d42016f44d539f4c03185a17cd64" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:13.340176 1316 raft_consensus.cc:513] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.341817 1273 ts_tablet_manager.cc:1428] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20250114 02:53:13.342012 1045 leader_election.cc:343] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.340432 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3144d42016f44d539f4c03185a17cd64" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:13.342746 1316 leader_election.cc:290] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:13.342798 1231 raft_consensus.cc:3054] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.343199 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb6d62d39fa47059bcf3d88f235c999" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34" is_pre_election: true
I20250114 02:53:13.343786 1081 raft_consensus.cc:2463] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 0.
I20250114 02:53:13.344817 1195 leader_election.cc:304] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:13.345742 1317 raft_consensus.cc:2798] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:13.345736 1271 tablet_bootstrap.cc:654] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.347245 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb6d62d39fa47059bcf3d88f235c999" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc" is_pre_election: true
W20250114 02:53:13.349175 1194 leader_election.cc:343] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.349920 1273 tablet_bootstrap.cc:492] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236: Bootstrap starting.
I20250114 02:53:13.350664 1317 raft_consensus.cc:491] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:13.357152 1317 raft_consensus.cc:3054] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.362425 1231 raft_consensus.cc:2463] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 1.
I20250114 02:53:13.363448 1047 leader_election.cc:304] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34; no voters: 678704d2559e41eb9d8b76af869bbefc
I20250114 02:53:13.364432 1277 raft_consensus.cc:2798] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:13.364830 1277 raft_consensus.cc:695] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Becoming Leader. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:13.365586 1277 consensus_queue.cc:237] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.372215 1272 tablet_bootstrap.cc:492] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc: No bootstrap required, opened a new log
I20250114 02:53:13.372608 1272 ts_tablet_manager.cc:1397] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc: Time spent bootstrapping tablet: real 0.088s user 0.002s sys 0.024s
I20250114 02:53:13.373865 1317 raft_consensus.cc:513] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.374754 1272 raft_consensus.cc:357] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.376780 1272 raft_consensus.cc:383] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.377033 1272 raft_consensus.cc:738] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.377760 1272 consensus_queue.cc:260] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.377379 1317 leader_election.cc:290] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:13.378520 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb6d62d39fa47059bcf3d88f235c999" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:13.379096 1081 raft_consensus.cc:3054] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.379864 1272 ts_tablet_manager.cc:1428] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20250114 02:53:13.380661 1272 tablet_bootstrap.cc:492] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc: Bootstrap starting.
I20250114 02:53:13.381096 1273 tablet_bootstrap.cc:654] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.385052 1081 raft_consensus.cc:2463] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 1.
I20250114 02:53:13.385797 1272 tablet_bootstrap.cc:654] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.386850 1195 leader_election.cc:304] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:13.388254 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb6d62d39fa47059bcf3d88f235c999" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:13.388923 1156 raft_consensus.cc:3054] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.388064 979 catalog_manager.cc:5526] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 reported cstate change: term changed from 0 to 1, leader changed from <none> to dba85f60dcc94b6797318aef42473236 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "dba85f60dcc94b6797318aef42473236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:13.391431 1273 tablet_bootstrap.cc:492] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236: No bootstrap required, opened a new log
I20250114 02:53:13.391834 1273 ts_tablet_manager.cc:1397] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.002s
I20250114 02:53:13.393987 1273 raft_consensus.cc:357] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.394438 1272 tablet_bootstrap.cc:492] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc: No bootstrap required, opened a new log
I20250114 02:53:13.394568 1273 raft_consensus.cc:383] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.394850 1272 ts_tablet_manager.cc:1397] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20250114 02:53:13.394943 1273 raft_consensus.cc:738] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.395587 1273 consensus_queue.cc:260] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.396482 1280 raft_consensus.cc:2798] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:13.396973 1280 raft_consensus.cc:695] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Becoming Leader. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:13.397631 1273 ts_tablet_manager.cc:1428] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:53:13.397363 1272 raft_consensus.cc:357] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.398126 1272 raft_consensus.cc:383] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.397718 1280 consensus_queue.cc:237] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.398401 1272 raft_consensus.cc:738] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.398579 1273 tablet_bootstrap.cc:492] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236: Bootstrap starting.
I20250114 02:53:13.399102 1272 consensus_queue.cc:260] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.401410 1272 ts_tablet_manager.cc:1428] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 02:53:13.402280 1316 raft_consensus.cc:491] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.402671 1316 raft_consensus.cc:513] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.404467 1273 tablet_bootstrap.cc:654] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.404376 1316 leader_election.cc:290] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:13.402280 1272 tablet_bootstrap.cc:492] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc: Bootstrap starting.
I20250114 02:53:13.406005 1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf4d4a188940c685b6726024522268" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc" is_pre_election: true
I20250114 02:53:13.406301 1080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf4d4a188940c685b6726024522268" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34" is_pre_election: true
I20250114 02:53:13.406648 1157 raft_consensus.cc:2463] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 0.
W20250114 02:53:13.407557 1195 leader_election.cc:343] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.404734 1156 raft_consensus.cc:2463] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 1.
I20250114 02:53:13.408764 1194 leader_election.cc:304] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters: fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:13.409579 1316 raft_consensus.cc:2798] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:13.409914 1316 raft_consensus.cc:491] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:13.410231 1316 raft_consensus.cc:3054] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.408143 1317 raft_consensus.cc:491] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.413130 1317 raft_consensus.cc:513] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.414126 1271 tablet_bootstrap.cc:492] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34: No bootstrap required, opened a new log
I20250114 02:53:13.414515 1271 ts_tablet_manager.cc:1397] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34: Time spent bootstrapping tablet: real 0.088s user 0.017s sys 0.019s
I20250114 02:53:13.414943 1317 leader_election.cc:290] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:13.416007 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34" is_pre_election: true
I20250114 02:53:13.416538 1081 raft_consensus.cc:2463] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 0.
I20250114 02:53:13.417155 1271 raft_consensus.cc:357] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.417362 1195 leader_election.cc:304] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:13.417750 1271 raft_consensus.cc:383] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.418004 1271 raft_consensus.cc:738] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.418067 1317 raft_consensus.cc:2798] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:13.418576 1317 raft_consensus.cc:491] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:13.418498 1271 consensus_queue.cc:260] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.419209 1317 raft_consensus.cc:3054] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.420730 1271 ts_tablet_manager.cc:1428] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:53:13.421499 1271 tablet_bootstrap.cc:492] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34: Bootstrap starting.
I20250114 02:53:13.423411 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc" is_pre_election: true
I20250114 02:53:13.423987 1156 raft_consensus.cc:2463] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 0.
I20250114 02:53:13.405109 977 catalog_manager.cc:5526] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65). New cstate: current_term: 1 leader_uuid: "fe3889e52a2c4bc2873b387533c90d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:13.430382 977 catalog_manager.cc:5526] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65). New cstate: current_term: 1 leader_uuid: "fe3889e52a2c4bc2873b387533c90d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:13.432328 977 catalog_manager.cc:5526] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65). New cstate: current_term: 1 leader_uuid: "fe3889e52a2c4bc2873b387533c90d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:13.430701 1316 raft_consensus.cc:513] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.436120 1316 leader_election.cc:290] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:13.436070 1317 raft_consensus.cc:513] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.439245 1317 leader_election.cc:290] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:13.440284 1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:13.440822 1157 raft_consensus.cc:3054] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.441020 1273 tablet_bootstrap.cc:492] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236: No bootstrap required, opened a new log
I20250114 02:53:13.441310 1271 tablet_bootstrap.cc:654] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.441399 1273 ts_tablet_manager.cc:1397] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236: Time spent bootstrapping tablet: real 0.043s user 0.016s sys 0.020s
I20250114 02:53:13.442885 1080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:13.443596 1080 raft_consensus.cc:3054] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.444100 1273 raft_consensus.cc:357] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.444808 1273 raft_consensus.cc:383] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.445092 1273 raft_consensus.cc:738] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.445242 1272 tablet_bootstrap.cc:654] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.445829 1273 consensus_queue.cc:260] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.447329 1157 raft_consensus.cc:2463] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 1.
I20250114 02:53:13.447467 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf4d4a188940c685b6726024522268" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:13.448244 1273 ts_tablet_manager.cc:1428] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250114 02:53:13.448347 1194 leader_election.cc:304] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters:
W20250114 02:53:13.448868 1195 leader_election.cc:343] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.449066 1317 raft_consensus.cc:2798] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:13.449599 1317 raft_consensus.cc:695] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Becoming Leader. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:13.447351 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf4d4a188940c685b6726024522268" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:13.450245 1156 raft_consensus.cc:3054] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.450452 1317 consensus_queue.cc:237] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.451335 1272 tablet_bootstrap.cc:492] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc: No bootstrap required, opened a new log
I20250114 02:53:13.451718 1272 ts_tablet_manager.cc:1397] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.000s
I20250114 02:53:13.454313 1272 raft_consensus.cc:357] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.455174 1272 raft_consensus.cc:383] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.455493 1272 raft_consensus.cc:738] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.456296 1272 consensus_queue.cc:260] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.458222 1313 raft_consensus.cc:491] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.458817 1313 raft_consensus.cc:513] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.462332 1313 leader_election.cc:290] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.462774 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3144d42016f44d539f4c03185a17cd64" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236" is_pre_election: true
I20250114 02:53:13.463783 1231 raft_consensus.cc:2388] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 678704d2559e41eb9d8b76af869bbefc in current term 1: Already voted for candidate fe3889e52a2c4bc2873b387533c90d34 in this term.
I20250114 02:53:13.464038 1272 ts_tablet_manager.cc:1428] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20250114 02:53:13.466360 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3144d42016f44d539f4c03185a17cd64" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34" is_pre_election: true
I20250114 02:53:13.468072 1121 leader_election.cc:304] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc; no voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:13.469175 1313 raft_consensus.cc:3054] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.471505 1080 raft_consensus.cc:2463] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 1.
I20250114 02:53:13.474354 1313 raft_consensus.cc:2743] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:13.480121 979 catalog_manager.cc:5526] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 reported cstate change: term changed from 0 to 1, leader changed from <none> to dba85f60dcc94b6797318aef42473236 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "dba85f60dcc94b6797318aef42473236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:13.485251 1156 raft_consensus.cc:2463] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 1.
I20250114 02:53:13.486413 1194 leader_election.cc:304] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters: fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:13.487397 1317 raft_consensus.cc:2798] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:13.487820 1317 raft_consensus.cc:695] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Becoming Leader. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:13.488814 1271 tablet_bootstrap.cc:492] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34: No bootstrap required, opened a new log
I20250114 02:53:13.488539 1317 consensus_queue.cc:237] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.489241 1271 ts_tablet_manager.cc:1397] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34: Time spent bootstrapping tablet: real 0.068s user 0.019s sys 0.027s
I20250114 02:53:13.491307 1271 raft_consensus.cc:357] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.491981 1271 raft_consensus.cc:383] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.492265 1271 raft_consensus.cc:738] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.492791 1271 consensus_queue.cc:260] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.494745 1271 ts_tablet_manager.cc:1428] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20250114 02:53:13.495514 1271 tablet_bootstrap.cc:492] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34: Bootstrap starting.
I20250114 02:53:13.498795 978 catalog_manager.cc:5526] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 reported cstate change: term changed from 0 to 1, leader changed from <none> to dba85f60dcc94b6797318aef42473236 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "dba85f60dcc94b6797318aef42473236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:13.500250 1271 tablet_bootstrap.cc:654] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:13.500515 978 catalog_manager.cc:5526] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 reported cstate change: term changed from 0 to 1, leader changed from <none> to dba85f60dcc94b6797318aef42473236 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "dba85f60dcc94b6797318aef42473236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:13.503378 1313 raft_consensus.cc:491] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.504094 1271 tablet_bootstrap.cc:492] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34: No bootstrap required, opened a new log
I20250114 02:53:13.503787 1313 raft_consensus.cc:513] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.504436 1271 ts_tablet_manager.cc:1397] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20250114 02:53:13.505456 1313 leader_election.cc:290] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.506300 1080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7415008b1fac4680894ed783f47e65fd" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34" is_pre_election: true
I20250114 02:53:13.506767 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7415008b1fac4680894ed783f47e65fd" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236" is_pre_election: true
I20250114 02:53:13.507318 1231 raft_consensus.cc:2463] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 0.
I20250114 02:53:13.507216 1271 raft_consensus.cc:357] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
W20250114 02:53:13.507629 1121 leader_election.cc:343] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:13.507791 1271 raft_consensus.cc:383] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:13.508016 1271 raft_consensus.cc:738] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Initialized, Role: FOLLOWER
I20250114 02:53:13.508167 1122 leader_election.cc:304] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters: fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:13.508756 1313 raft_consensus.cc:2798] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:13.508512 1271 consensus_queue.cc:260] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.509099 1313 raft_consensus.cc:491] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:13.509416 1313 raft_consensus.cc:3054] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.510493 1271 ts_tablet_manager.cc:1428] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:53:13.514089 1313 raft_consensus.cc:513] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.515394 1313 leader_election.cc:290] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.515909 1080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7415008b1fac4680894ed783f47e65fd" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:13.516184 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7415008b1fac4680894ed783f47e65fd" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:13.516431 1080 raft_consensus.cc:3054] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.516671 1231 raft_consensus.cc:3054] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.519925 1313 raft_consensus.cc:491] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.520299 1313 raft_consensus.cc:513] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.521672 1313 leader_election.cc:290] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.522325 1080 raft_consensus.cc:2463] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 1.
I20250114 02:53:13.522329 1231 raft_consensus.cc:2463] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 1.
I20250114 02:53:13.522594 1230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3949440ca34a658306d0065161a875" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236" is_pre_election: true
I20250114 02:53:13.523000 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3949440ca34a658306d0065161a875" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34" is_pre_election: true
I20250114 02:53:13.523196 1230 raft_consensus.cc:2463] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 0.
I20250114 02:53:13.523350 1122 leader_election.cc:304] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters:
I20250114 02:53:13.523579 1081 raft_consensus.cc:2463] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 0.
I20250114 02:53:13.524070 1313 raft_consensus.cc:2798] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:13.524225 1122 leader_election.cc:304] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters:
I20250114 02:53:13.524989 1311 raft_consensus.cc:2798] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:13.525230 1313 raft_consensus.cc:695] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 1 LEADER]: Becoming Leader. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Running, Role: LEADER
I20250114 02:53:13.525308 1311 raft_consensus.cc:491] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:13.525708 1311 raft_consensus.cc:3054] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.525938 1313 consensus_queue.cc:237] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.531018 1311 raft_consensus.cc:513] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.532321 979 catalog_manager.cc:5526] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc reported cstate change: term changed from 0 to 1, leader changed from <none> to 678704d2559e41eb9d8b76af869bbefc (127.0.106.66). New cstate: current_term: 1 leader_uuid: "678704d2559e41eb9d8b76af869bbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:13.533001 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3949440ca34a658306d0065161a875" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:13.533217 1230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3949440ca34a658306d0065161a875" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:13.533468 1081 raft_consensus.cc:3054] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.533793 1230 raft_consensus.cc:3054] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.532652 1311 leader_election.cc:290] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.538323 1081 raft_consensus.cc:2463] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 1.
I20250114 02:53:13.538738 1230 raft_consensus.cc:2463] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 1.
I20250114 02:53:13.539189 1121 leader_election.cc:304] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:13.540069 1311 raft_consensus.cc:2798] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:13.540413 1311 raft_consensus.cc:695] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 1 LEADER]: Becoming Leader. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Running, Role: LEADER
I20250114 02:53:13.541016 1311 consensus_queue.cc:237] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.547416 979 catalog_manager.cc:5526] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc reported cstate change: term changed from 0 to 1, leader changed from <none> to 678704d2559e41eb9d8b76af869bbefc (127.0.106.66). New cstate: current_term: 1 leader_uuid: "678704d2559e41eb9d8b76af869bbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:13.614439 1277 raft_consensus.cc:491] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.614908 1277 raft_consensus.cc:513] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.616497 1277 leader_election.cc:290] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.617208 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf4d4a188940c685b6726024522268" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc" is_pre_election: true
I20250114 02:53:13.617405 1230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf4d4a188940c685b6726024522268" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba85f60dcc94b6797318aef42473236" is_pre_election: true
I20250114 02:53:13.617792 1156 raft_consensus.cc:2388] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3889e52a2c4bc2873b387533c90d34 in current term 1: Already voted for candidate dba85f60dcc94b6797318aef42473236 in this term.
I20250114 02:53:13.618590 1045 leader_election.cc:304] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3889e52a2c4bc2873b387533c90d34; no voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236
I20250114 02:53:13.619153 1277 raft_consensus.cc:3054] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.622965 1277 raft_consensus.cc:2743] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:13.628309 1277 consensus_queue.cc:1035] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.639067 1277 consensus_queue.cc:1035] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.659394 1277 consensus_queue.cc:1035] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.667919 1306 consensus_queue.cc:1035] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:13.719990 1317 raft_consensus.cc:491] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:13.720424 1317 raft_consensus.cc:513] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.721836 1317 leader_election.cc:290] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:13.722571 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3889e52a2c4bc2873b387533c90d34" is_pre_election: true
I20250114 02:53:13.722750 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678704d2559e41eb9d8b76af869bbefc" is_pre_election: true
I20250114 02:53:13.723387 1156 raft_consensus.cc:2388] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dba85f60dcc94b6797318aef42473236 in current term 1: Already voted for candidate fe3889e52a2c4bc2873b387533c90d34 in this term.
I20250114 02:53:13.724148 1194 leader_election.cc:304] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dba85f60dcc94b6797318aef42473236; no voters: 678704d2559e41eb9d8b76af869bbefc, fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:13.724690 1317 raft_consensus.cc:3054] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:13.728824 1317 raft_consensus.cc:2743] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:13.735858 1306 consensus_queue.cc:1035] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.744213 1306 consensus_queue.cc:1035] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.779254 1156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b3949440ca34a658306d0065161a875"
dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
mode: GRACEFUL
new_leader_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:43966
I20250114 02:53:13.779798 1156 raft_consensus.cc:604] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 1 LEADER]: Received request to transfer leadership to TS fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:13.795339 1317 consensus_queue.cc:1035] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.795656 1230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722"
dest_uuid: "dba85f60dcc94b6797318aef42473236"
mode: GRACEFUL
new_leader_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:38168
I20250114 02:53:13.796265 1230 raft_consensus.cc:604] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Received request to transfer leadership to TS fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:13.801138 1230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0"
dest_uuid: "dba85f60dcc94b6797318aef42473236"
mode: GRACEFUL
new_leader_uuid: "678704d2559e41eb9d8b76af869bbefc"
from {username='slave'} at 127.0.0.1:38168
I20250114 02:53:13.801898 1230 raft_consensus.cc:604] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Received request to transfer leadership to TS 678704d2559e41eb9d8b76af869bbefc
I20250114 02:53:13.807235 1230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bfb6d62d39fa47059bcf3d88f235c999"
dest_uuid: "dba85f60dcc94b6797318aef42473236"
mode: GRACEFUL
new_leader_uuid: "678704d2559e41eb9d8b76af869bbefc"
from {username='slave'} at 127.0.0.1:38168
I20250114 02:53:13.808578 1230 raft_consensus.cc:604] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Received request to transfer leadership to TS 678704d2559e41eb9d8b76af869bbefc
I20250114 02:53:13.808938 1317 consensus_queue.cc:1035] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.823004 1316 raft_consensus.cc:988] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236: : Instructing follower fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:13.823387 1316 raft_consensus.cc:1076] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Signalling peer fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:13.824999 1080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:44686
I20250114 02:53:13.825582 1080 raft_consensus.cc:491] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:13.825865 1080 raft_consensus.cc:3054] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:13.830286 1080 raft_consensus.cc:513] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.832108 1080 leader_election.cc:290] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 2 election: Requested vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.832778 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:13.833389 1156 raft_consensus.cc:3054] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:13.834722 1230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:13.835268 1230 raft_consensus.cc:3049] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:13.835304 1080 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
mode: GRACEFUL
new_leader_uuid: "678704d2559e41eb9d8b76af869bbefc"
from {username='slave'} at 127.0.0.1:55022
I20250114 02:53:13.835635 1230 raft_consensus.cc:738] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:13.835922 1080 raft_consensus.cc:604] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Received request to transfer leadership to TS 678704d2559e41eb9d8b76af869bbefc
I20250114 02:53:13.836357 1230 consensus_queue.cc:260] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.837529 1230 raft_consensus.cc:3054] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:13.838320 1156 raft_consensus.cc:2463] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 2.
I20250114 02:53:13.839231 1045 leader_election.cc:304] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:13.840003 1306 raft_consensus.cc:2798] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:13.840567 1306 raft_consensus.cc:695] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 2 LEADER]: Becoming Leader. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:13.841562 1306 consensus_queue.cc:237] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.843240 1230 raft_consensus.cc:2463] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 2.
I20250114 02:53:13.843870 1231 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bebf4d4a188940c685b6726024522268"
dest_uuid: "dba85f60dcc94b6797318aef42473236"
mode: GRACEFUL
new_leader_uuid: "678704d2559e41eb9d8b76af869bbefc"
from {username='slave'} at 127.0.0.1:38168
I20250114 02:53:13.844489 1231 raft_consensus.cc:604] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Received request to transfer leadership to TS 678704d2559e41eb9d8b76af869bbefc
I20250114 02:53:13.850595 1156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7415008b1fac4680894ed783f47e65fd"
dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
mode: GRACEFUL
new_leader_uuid: "dba85f60dcc94b6797318aef42473236"
from {username='slave'} at 127.0.0.1:43966
I20250114 02:53:13.851192 1156 raft_consensus.cc:604] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 1 LEADER]: Received request to transfer leadership to TS dba85f60dcc94b6797318aef42473236
I20250114 02:53:13.851024 979 catalog_manager.cc:5526] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 reported cstate change: term changed from 1 to 2, leader changed from dba85f60dcc94b6797318aef42473236 (127.0.106.67) to fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65). New cstate: current_term: 2 leader_uuid: "fe3889e52a2c4bc2873b387533c90d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:13.874912 1306 consensus_queue.cc:1035] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.874912 1317 consensus_queue.cc:1035] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.887038 1317 consensus_queue.cc:1035] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.889181 1305 consensus_queue.cc:1035] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.896071 1316 consensus_queue.cc:1035] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.901044 1316 raft_consensus.cc:988] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236: : Instructing follower 678704d2559e41eb9d8b76af869bbefc to start an election
I20250114 02:53:13.901504 1316 raft_consensus.cc:1076] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Signalling peer 678704d2559e41eb9d8b76af869bbefc to start an election
I20250114 02:53:13.903761 1157 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bfb6d62d39fa47059bcf3d88f235c999"
dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
from {username='slave'} at 127.0.0.1:32904
I20250114 02:53:13.904264 1157 raft_consensus.cc:491] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:13.904608 1157 raft_consensus.cc:3054] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:13.905401 1280 raft_consensus.cc:988] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236: : Instructing follower 678704d2559e41eb9d8b76af869bbefc to start an election
I20250114 02:53:13.905829 1280 raft_consensus.cc:1076] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Signalling peer 678704d2559e41eb9d8b76af869bbefc to start an election
I20250114 02:53:13.907455 1154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0"
dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
from {username='slave'} at 127.0.0.1:32904
I20250114 02:53:13.908336 1154 raft_consensus.cc:491] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:13.908735 1154 raft_consensus.cc:3054] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:13.910856 1157 raft_consensus.cc:513] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.913494 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb6d62d39fa47059bcf3d88f235c999" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:13.914037 1081 raft_consensus.cc:3054] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:13.915220 1280 consensus_queue.cc:1035] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:13.915262 1154 raft_consensus.cc:513] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.917399 1154 leader_election.cc:290] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 2 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.920099 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:13.920640 1231 raft_consensus.cc:3049] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:13.920990 1231 raft_consensus.cc:738] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:13.921767 1231 consensus_queue.cc:260] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.923429 1082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:13.924917 1230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb6d62d39fa47059bcf3d88f235c999" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:13.925534 1230 raft_consensus.cc:3049] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:13.923449 1231 raft_consensus.cc:3054] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:13.925881 1230 raft_consensus.cc:738] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:13.926776 1230 consensus_queue.cc:260] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.927834 1230 raft_consensus.cc:3054] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:13.930142 1231 raft_consensus.cc:2463] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 2.
I20250114 02:53:13.931739 1081 raft_consensus.cc:2463] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 2.
I20250114 02:53:13.932911 1121 leader_election.cc:304] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:13.933481 1311 raft_consensus.cc:2798] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:13.933948 1311 raft_consensus.cc:695] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Becoming Leader. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Running, Role: LEADER
I20250114 02:53:13.934063 1122 leader_election.cc:304] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters: fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:13.934870 1157 leader_election.cc:290] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 2 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:13.935490 1313 raft_consensus.cc:2798] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:13.934759 1311 consensus_queue.cc:237] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.936020 1313 raft_consensus.cc:695] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Becoming Leader. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Running, Role: LEADER
I20250114 02:53:13.936828 1313 consensus_queue.cc:237] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.943610 1230 raft_consensus.cc:2463] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 2.
I20250114 02:53:13.948073 979 catalog_manager.cc:5526] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc reported cstate change: term changed from 1 to 2, leader changed from dba85f60dcc94b6797318aef42473236 (127.0.106.67) to 678704d2559e41eb9d8b76af869bbefc (127.0.106.66). New cstate: current_term: 2 leader_uuid: "678704d2559e41eb9d8b76af869bbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:13.949098 979 catalog_manager.cc:5526] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc reported cstate change: term changed from 1 to 2, leader changed from dba85f60dcc94b6797318aef42473236 (127.0.106.67) to 678704d2559e41eb9d8b76af869bbefc (127.0.106.66). New cstate: current_term: 2 leader_uuid: "678704d2559e41eb9d8b76af869bbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:13.970026 1358 consensus_queue.cc:1035] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.980446 1360 consensus_queue.cc:1035] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:13.986773 1311 raft_consensus.cc:988] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc: : Instructing follower dba85f60dcc94b6797318aef42473236 to start an election
I20250114 02:53:13.987216 1360 raft_consensus.cc:1076] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 1 LEADER]: Signalling peer dba85f60dcc94b6797318aef42473236 to start an election
I20250114 02:53:13.988600 1230 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7415008b1fac4680894ed783f47e65fd"
dest_uuid: "dba85f60dcc94b6797318aef42473236"
from {username='slave'} at 127.0.0.1:37590
I20250114 02:53:13.989019 1230 raft_consensus.cc:491] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:13.989274 1230 raft_consensus.cc:3054] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:13.993511 1230 raft_consensus.cc:513] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.995076 1230 leader_election.cc:290] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 2 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:13.995666 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7415008b1fac4680894ed783f47e65fd" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:13.996021 1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7415008b1fac4680894ed783f47e65fd" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:13.996240 1081 raft_consensus.cc:3054] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:13.996600 1157 raft_consensus.cc:3049] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:13.996841 1157 raft_consensus.cc:738] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Running, Role: LEADER
I20250114 02:53:13.997450 1157 consensus_queue.cc:260] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:13.998271 1157 raft_consensus.cc:3054] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:14.001749 1081 raft_consensus.cc:2463] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 2.
I20250114 02:53:14.002658 1195 leader_election.cc:304] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:14.003046 1157 raft_consensus.cc:2463] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 2.
I20250114 02:53:14.003347 1316 raft_consensus.cc:2798] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:14.003793 1316 raft_consensus.cc:695] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [term 2 LEADER]: Becoming Leader. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:14.004410 1316 consensus_queue.cc:237] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:14.008877 1316 consensus_queue.cc:1035] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:14.011250 977 catalog_manager.cc:5526] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 reported cstate change: term changed from 1 to 2, leader changed from 678704d2559e41eb9d8b76af869bbefc (127.0.106.66) to dba85f60dcc94b6797318aef42473236 (127.0.106.67). New cstate: current_term: 2 leader_uuid: "dba85f60dcc94b6797318aef42473236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:14.020766 1317 consensus_queue.cc:1035] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:14.027348 1311 consensus_queue.cc:1035] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:14.029042 1280 raft_consensus.cc:988] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236: : Instructing follower 678704d2559e41eb9d8b76af869bbefc to start an election
I20250114 02:53:14.029477 1280 raft_consensus.cc:1076] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Signalling peer 678704d2559e41eb9d8b76af869bbefc to start an election
I20250114 02:53:14.031095 1157 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bebf4d4a188940c685b6726024522268"
dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
from {username='slave'} at 127.0.0.1:32904
I20250114 02:53:14.031600 1157 raft_consensus.cc:491] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:14.031955 1157 raft_consensus.cc:3054] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:14.035174 1360 raft_consensus.cc:988] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc: : Instructing follower fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:14.035619 1358 raft_consensus.cc:1076] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 1 LEADER]: Signalling peer fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:14.037691 1080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b3949440ca34a658306d0065161a875"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:44688
I20250114 02:53:14.038211 1080 raft_consensus.cc:491] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:14.038450 1080 raft_consensus.cc:3054] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:14.040014 1157 raft_consensus.cc:513] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:14.041334 1313 consensus_queue.cc:1035] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:14.045043 1157 leader_election.cc:290] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 2 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:14.045262 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf4d4a188940c685b6726024522268" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:14.044881 1080 raft_consensus.cc:513] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:14.045818 1231 raft_consensus.cc:3049] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:14.046062 1231 raft_consensus.cc:738] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:14.046813 1231 consensus_queue.cc:260] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:14.047845 1231 raft_consensus.cc:3054] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:14.051111 1080 leader_election.cc:290] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 2 election: Requested vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:14.056170 1229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3949440ca34a658306d0065161a875" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:14.057538 1231 raft_consensus.cc:2463] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 2.
I20250114 02:53:14.058396 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebf4d4a188940c685b6726024522268" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:14.058990 1081 raft_consensus.cc:3054] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:14.060185 1122 leader_election.cc:304] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters:
I20250114 02:53:14.061030 1313 raft_consensus.cc:2798] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:14.061447 1313 raft_consensus.cc:695] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Becoming Leader. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Running, Role: LEADER
I20250114 02:53:14.062182 1313 consensus_queue.cc:237] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:14.065044 1081 raft_consensus.cc:2463] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 2.
I20250114 02:53:14.066143 1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3949440ca34a658306d0065161a875" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:14.066968 1157 raft_consensus.cc:3049] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:14.067296 1157 raft_consensus.cc:738] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Running, Role: LEADER
I20250114 02:53:14.067962 1157 consensus_queue.cc:260] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:14.069296 1157 raft_consensus.cc:3054] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:14.069713 978 catalog_manager.cc:5526] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc reported cstate change: term changed from 1 to 2, leader changed from dba85f60dcc94b6797318aef42473236 (127.0.106.67) to 678704d2559e41eb9d8b76af869bbefc (127.0.106.66). New cstate: current_term: 2 leader_uuid: "678704d2559e41eb9d8b76af869bbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:14.074541 1157 raft_consensus.cc:2463] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 2.
I20250114 02:53:14.075454 1045 leader_election.cc:304] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, fe3889e52a2c4bc2873b387533c90d34; no voters: dba85f60dcc94b6797318aef42473236
I20250114 02:53:14.076182 1307 raft_consensus.cc:2798] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:14.076640 1307 raft_consensus.cc:695] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 2 LEADER]: Becoming Leader. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:14.077203 1307 consensus_queue.cc:237] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:14.082353 978 catalog_manager.cc:5526] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 reported cstate change: term changed from 1 to 2, leader changed from 678704d2559e41eb9d8b76af869bbefc (127.0.106.66) to fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65). New cstate: current_term: 2 leader_uuid: "fe3889e52a2c4bc2873b387533c90d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:14.144163 1277 raft_consensus.cc:988] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34: : Instructing follower 678704d2559e41eb9d8b76af869bbefc to start an election
I20250114 02:53:14.144534 1307 raft_consensus.cc:1076] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Signalling peer 678704d2559e41eb9d8b76af869bbefc to start an election
I20250114 02:53:14.145758 1157 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89"
dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
from {username='slave'} at 127.0.0.1:32890
I20250114 02:53:14.146169 1157 raft_consensus.cc:491] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:14.146401 1157 raft_consensus.cc:3054] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:14.150540 1157 raft_consensus.cc:513] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:14.151957 1157 leader_election.cc:290] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 2 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:14.152547 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:14.153045 1081 raft_consensus.cc:3049] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:14.153108 1230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:14.153295 1081 raft_consensus.cc:738] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:14.153676 1230 raft_consensus.cc:3054] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:14.154017 1081 consensus_queue.cc:260] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:14.155301 1081 raft_consensus.cc:3054] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:14.157531 1230 raft_consensus.cc:2463] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 2.
I20250114 02:53:14.158360 1122 leader_election.cc:304] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters:
I20250114 02:53:14.158891 1358 raft_consensus.cc:2798] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:14.158929 1081 raft_consensus.cc:2463] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 2.
I20250114 02:53:14.159462 1358 raft_consensus.cc:695] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Becoming Leader. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Running, Role: LEADER
I20250114 02:53:14.160224 1358 consensus_queue.cc:237] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:14.165717 978 catalog_manager.cc:5526] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc reported cstate change: term changed from 1 to 2, leader changed from fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65) to 678704d2559e41eb9d8b76af869bbefc (127.0.106.66). New cstate: current_term: 2 leader_uuid: "678704d2559e41eb9d8b76af869bbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:14.280849 1230 raft_consensus.cc:1270] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.281956 1307 consensus_queue.cc:1035] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:14.288463 1157 raft_consensus.cc:1270] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.289752 1277 consensus_queue.cc:1035] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:14.322682 1230 raft_consensus.cc:1270] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Refusing update from remote peer 678704d2559e41eb9d8b76af869bbefc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.323696 1358 consensus_queue.cc:1035] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:14.329898 1081 raft_consensus.cc:1270] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Refusing update from remote peer 678704d2559e41eb9d8b76af869bbefc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.331220 1360 consensus_queue.cc:1035] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:14.369267 1081 raft_consensus.cc:3054] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:14.374658 1081 raft_consensus.cc:1270] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Refusing update from remote peer 678704d2559e41eb9d8b76af869bbefc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.375550 1360 consensus_queue.cc:1035] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:14.381290 1230 raft_consensus.cc:1270] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Refusing update from remote peer 678704d2559e41eb9d8b76af869bbefc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.382686 1358 consensus_queue.cc:1035] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:14.402791 1081 raft_consensus.cc:1270] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Refusing update from remote peer dba85f60dcc94b6797318aef42473236: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.403793 1317 consensus_queue.cc:1035] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:14.410210 1157 raft_consensus.cc:1270] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Refusing update from remote peer dba85f60dcc94b6797318aef42473236: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.411840 1317 consensus_queue.cc:1035] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:14.469347 1157 raft_consensus.cc:1270] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.470503 1277 consensus_queue.cc:1035] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:14.477226 1230 raft_consensus.cc:3054] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:14.480412 1231 raft_consensus.cc:1270] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Refusing update from remote peer 678704d2559e41eb9d8b76af869bbefc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.481688 1358 consensus_queue.cc:1035] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:14.483529 1230 raft_consensus.cc:1270] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.485082 1277 consensus_queue.cc:1035] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:14.488344 1081 raft_consensus.cc:1270] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Refusing update from remote peer 678704d2559e41eb9d8b76af869bbefc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.491240 1313 consensus_queue.cc:1035] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:14.543933 1231 raft_consensus.cc:1270] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Refusing update from remote peer 678704d2559e41eb9d8b76af869bbefc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.544844 1313 consensus_queue.cc:1035] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:14.552124 1081 raft_consensus.cc:1270] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Refusing update from remote peer 678704d2559e41eb9d8b76af869bbefc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:14.553426 1313 consensus_queue.cc:1035] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:16.875536 425 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20250114 02:53:16.876142 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:16.889947 1081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b3949440ca34a658306d0065161a875"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
mode: GRACEFUL
new_leader_uuid: "dba85f60dcc94b6797318aef42473236"
from {username='slave'} at 127.0.0.1:55026
I20250114 02:53:16.890420 1081 raft_consensus.cc:604] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 2 LEADER]: Received request to transfer leadership to TS dba85f60dcc94b6797318aef42473236
I20250114 02:53:16.892410 1081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
mode: GRACEFUL
new_leader_uuid: "dba85f60dcc94b6797318aef42473236"
from {username='slave'} at 127.0.0.1:55026
I20250114 02:53:16.892915 1081 raft_consensus.cc:604] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 2 LEADER]: Received request to transfer leadership to TS dba85f60dcc94b6797318aef42473236
I20250114 02:53:16.893857 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250114 02:53:16.894232 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:17.171128 1306 raft_consensus.cc:988] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34: : Instructing follower dba85f60dcc94b6797318aef42473236 to start an election
I20250114 02:53:17.171547 1371 raft_consensus.cc:1076] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 2 LEADER]: Signalling peer dba85f60dcc94b6797318aef42473236 to start an election
I20250114 02:53:17.172902 1230 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b3949440ca34a658306d0065161a875"
dest_uuid: "dba85f60dcc94b6797318aef42473236"
from {username='slave'} at 127.0.0.1:37588
I20250114 02:53:17.173310 1230 raft_consensus.cc:491] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:17.173557 1230 raft_consensus.cc:3054] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:17.178071 1230 raft_consensus.cc:513] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:17.179926 1230 leader_election.cc:290] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 3 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:17.181121 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3949440ca34a658306d0065161a875" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:17.181603 1156 raft_consensus.cc:3054] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:17.182399 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3949440ca34a658306d0065161a875" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:17.182963 1081 raft_consensus.cc:3049] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:17.183264 1081 raft_consensus.cc:738] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:17.183972 1081 consensus_queue.cc:260] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:17.185041 1081 raft_consensus.cc:3054] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:17.187654 1156 raft_consensus.cc:2463] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 3.
I20250114 02:53:17.188680 1194 leader_election.cc:304] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters:
I20250114 02:53:17.189277 1368 raft_consensus.cc:2798] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:17.190172 1081 raft_consensus.cc:2463] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 3.
I20250114 02:53:17.190264 1368 raft_consensus.cc:695] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 3 LEADER]: Becoming Leader. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:17.191099 1368 consensus_queue.cc:237] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:17.197252 979 catalog_manager.cc:5526] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 reported cstate change: term changed from 2 to 3, leader changed from fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65) to dba85f60dcc94b6797318aef42473236 (127.0.106.67). New cstate: current_term: 3 leader_uuid: "dba85f60dcc94b6797318aef42473236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:17.313011 1306 raft_consensus.cc:988] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34: : Instructing follower dba85f60dcc94b6797318aef42473236 to start an election
I20250114 02:53:17.313386 1371 raft_consensus.cc:1076] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 2 LEADER]: Signalling peer dba85f60dcc94b6797318aef42473236 to start an election
I20250114 02:53:17.314640 1230 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722"
dest_uuid: "dba85f60dcc94b6797318aef42473236"
from {username='slave'} at 127.0.0.1:37588
I20250114 02:53:17.315040 1230 raft_consensus.cc:491] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:17.315292 1230 raft_consensus.cc:3054] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:17.319392 1230 raft_consensus.cc:513] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:17.320698 1230 leader_election.cc:290] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 3 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:17.321373 1081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:17.321516 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:17.321771 1081 raft_consensus.cc:3049] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:17.321988 1156 raft_consensus.cc:3054] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:17.322031 1081 raft_consensus.cc:738] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:17.322717 1081 consensus_queue.cc:260] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:17.323500 1081 raft_consensus.cc:3054] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:17.325803 1156 raft_consensus.cc:2463] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 3.
I20250114 02:53:17.326597 1194 leader_election.cc:304] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters:
I20250114 02:53:17.327128 1368 raft_consensus.cc:2798] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:17.327539 1368 raft_consensus.cc:695] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 3 LEADER]: Becoming Leader. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:17.327550 1081 raft_consensus.cc:2463] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 3.
I20250114 02:53:17.328353 1368 consensus_queue.cc:237] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:17.334404 979 catalog_manager.cc:5526] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 reported cstate change: term changed from 2 to 3, leader changed from fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65) to dba85f60dcc94b6797318aef42473236 (127.0.106.67). New cstate: current_term: 3 leader_uuid: "dba85f60dcc94b6797318aef42473236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:17.597294 1156 raft_consensus.cc:1270] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Refusing update from remote peer dba85f60dcc94b6797318aef42473236: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:53:17.598332 1368 consensus_queue.cc:1035] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:53:17.607988 1081 raft_consensus.cc:1270] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Refusing update from remote peer dba85f60dcc94b6797318aef42473236: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:53:17.609228 1372 consensus_queue.cc:1035] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 02:53:17.741254 1156 raft_consensus.cc:1270] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Refusing update from remote peer dba85f60dcc94b6797318aef42473236: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:53:17.742288 1368 consensus_queue.cc:1035] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:53:17.751734 1081 raft_consensus.cc:1270] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Refusing update from remote peer dba85f60dcc94b6797318aef42473236: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:53:17.752907 1381 consensus_queue.cc:1035] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:53:17.895033 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:17.900591 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:17.900914 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250114 02:53:18.993189 1252 debug-util.cc:398] Leaking SignalData structure 0x7b08000efaa0 after lost signal to thread 428
W20250114 02:53:18.993922 1252 debug-util.cc:398] Leaking SignalData structure 0x7b08001dfec0 after lost signal to thread 1014
W20250114 02:53:18.994707 1252 debug-util.cc:398] Leaking SignalData structure 0x7b08001af2e0 after lost signal to thread 1106
W20250114 02:53:18.995361 1252 debug-util.cc:398] Leaking SignalData structure 0x7b08000a3260 after lost signal to thread 1181
W20250114 02:53:18.995993 1252 debug-util.cc:398] Leaking SignalData structure 0x7b08001de580 after lost signal to thread 1255
I20250114 02:53:19.081530 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:19.087529 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:19.087848 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:20.088557 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:20.093878 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:20.094291 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:21.095108 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:21.101718 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:21.102192 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:22.103068 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:22.107959 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:22.108284 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:23.109045 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:23.114526 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:23.114914 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:24.115587 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:24.120466 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:24.120817 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:25.121613 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:25.126538 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:25.126926 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:26.127632 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:26.133163 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:26.133539 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:27.157858 425 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20250114 02:53:27.163605 1231 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b3949440ca34a658306d0065161a875"
dest_uuid: "dba85f60dcc94b6797318aef42473236"
mode: GRACEFUL
new_leader_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:38168
I20250114 02:53:27.164203 1231 raft_consensus.cc:604] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 3 LEADER]: Received request to transfer leadership to TS fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:27.172032 1231 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722"
dest_uuid: "dba85f60dcc94b6797318aef42473236"
mode: GRACEFUL
new_leader_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:38168
I20250114 02:53:27.172430 1231 raft_consensus.cc:604] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 3 LEADER]: Received request to transfer leadership to TS fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:27.176491 1156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0"
dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
mode: GRACEFUL
new_leader_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:43966
I20250114 02:53:27.176970 1156 raft_consensus.cc:604] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Received request to transfer leadership to TS fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:27.180039 1395 raft_consensus.cc:988] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236: : Instructing follower fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:27.180624 1368 raft_consensus.cc:1076] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 3 LEADER]: Signalling peer fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:27.181777 1156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bfb6d62d39fa47059bcf3d88f235c999"
dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
mode: GRACEFUL
new_leader_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:43966
I20250114 02:53:27.181960 1080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:44686
I20250114 02:53:27.182166 1156 raft_consensus.cc:604] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Received request to transfer leadership to TS fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:27.182387 1080 raft_consensus.cc:491] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:27.182853 1080 raft_consensus.cc:3054] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:27.187901 1080 raft_consensus.cc:513] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.189815 1080 leader_election.cc:290] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 4 election: Requested vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:27.190110 1156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89"
dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
mode: GRACEFUL
new_leader_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:43966
I20250114 02:53:27.190707 1156 raft_consensus.cc:604] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Received request to transfer leadership to TS fe3889e52a2c4bc2873b387533c90d34
I20250114 02:53:27.190639 1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:27.191164 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc1d29533f046f9997fc1dd89c84722" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:27.191532 1157 raft_consensus.cc:3054] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:27.191649 1231 raft_consensus.cc:3049] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:53:27.191960 1231 raft_consensus.cc:738] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:27.192637 1231 consensus_queue.cc:260] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.193761 1231 raft_consensus.cc:3054] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:27.198292 1157 raft_consensus.cc:2463] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 4.
I20250114 02:53:27.199505 1045 leader_election.cc:304] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:27.200176 1231 raft_consensus.cc:2463] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 4.
I20250114 02:53:27.200251 1392 raft_consensus.cc:2798] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:53:27.200500 1396 raft_consensus.cc:988] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc: : Instructing follower fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:27.201078 1360 raft_consensus.cc:1076] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Signalling peer fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:27.201750 1392 raft_consensus.cc:695] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 4 LEADER]: Becoming Leader. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:27.202332 1080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bfb6d62d39fa47059bcf3d88f235c999"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:44688
I20250114 02:53:27.202726 1080 raft_consensus.cc:491] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:27.203045 1080 raft_consensus.cc:3054] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:27.202689 1392 consensus_queue.cc:237] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.208453 1080 raft_consensus.cc:513] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.210448 1080 leader_election.cc:290] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 3 election: Requested vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:27.211084 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb6d62d39fa47059bcf3d88f235c999" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:27.211380 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb6d62d39fa47059bcf3d88f235c999" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:27.211704 1156 raft_consensus.cc:3049] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:27.211872 1231 raft_consensus.cc:3054] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:27.212028 1156 raft_consensus.cc:738] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Running, Role: LEADER
I20250114 02:53:27.212746 1156 consensus_queue.cc:260] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.213724 1156 raft_consensus.cc:3054] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:27.217252 1231 raft_consensus.cc:2463] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 3.
I20250114 02:53:27.218097 1047 leader_election.cc:304] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:27.217495 979 catalog_manager.cc:5526] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 reported cstate change: term changed from 3 to 4, leader changed from dba85f60dcc94b6797318aef42473236 (127.0.106.67) to fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65). New cstate: current_term: 4 leader_uuid: "fe3889e52a2c4bc2873b387533c90d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:27.218704 1392 raft_consensus.cc:2798] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:27.219085 1392 raft_consensus.cc:695] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 3 LEADER]: Becoming Leader. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:27.219430 1156 raft_consensus.cc:2463] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 3.
I20250114 02:53:27.219894 1392 consensus_queue.cc:237] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.233209 979 catalog_manager.cc:5526] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 reported cstate change: term changed from 2 to 3, leader changed from 678704d2559e41eb9d8b76af869bbefc (127.0.106.66) to fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65). New cstate: current_term: 3 leader_uuid: "fe3889e52a2c4bc2873b387533c90d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:27.477594 1396 raft_consensus.cc:988] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc: : Instructing follower fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:27.477967 1360 raft_consensus.cc:1076] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Signalling peer fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:27.479058 1080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:44688
I20250114 02:53:27.479420 1080 raft_consensus.cc:491] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:27.479651 1080 raft_consensus.cc:3054] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:27.483382 1080 raft_consensus.cc:513] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.484655 1080 leader_election.cc:290] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 3 election: Requested vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:27.485188 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:27.485551 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:27.485756 1156 raft_consensus.cc:3049] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:27.485993 1231 raft_consensus.cc:3054] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:27.486017 1156 raft_consensus.cc:738] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Running, Role: LEADER
I20250114 02:53:27.486733 1156 consensus_queue.cc:260] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.487726 1156 raft_consensus.cc:3054] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:27.489765 1231 raft_consensus.cc:2463] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 3.
I20250114 02:53:27.490553 1047 leader_election.cc:304] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:27.491338 1397 raft_consensus.cc:2798] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:27.491678 1397 raft_consensus.cc:695] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 3 LEADER]: Becoming Leader. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:27.491787 1156 raft_consensus.cc:2463] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 3.
I20250114 02:53:27.492444 1397 consensus_queue.cc:237] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.498364 979 catalog_manager.cc:5526] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 reported cstate change: term changed from 2 to 3, leader changed from 678704d2559e41eb9d8b76af869bbefc (127.0.106.66) to fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65). New cstate: current_term: 3 leader_uuid: "fe3889e52a2c4bc2873b387533c90d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:27.505924 1395 raft_consensus.cc:988] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236: : Instructing follower fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:27.506219 1368 raft_consensus.cc:1076] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 3 LEADER]: Signalling peer fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:27.507172 1080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b3949440ca34a658306d0065161a875"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:44686
I20250114 02:53:27.507468 1080 raft_consensus.cc:491] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:27.507656 1080 raft_consensus.cc:3054] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:27.510902 1080 raft_consensus.cc:513] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.511910 1080 leader_election.cc:290] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 4 election: Requested vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:27.512485 1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3949440ca34a658306d0065161a875" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:27.512620 1231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3949440ca34a658306d0065161a875" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:27.512884 1156 raft_consensus.cc:3054] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:27.513000 1231 raft_consensus.cc:3049] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:53:27.513294 1231 raft_consensus.cc:738] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:27.513778 1231 consensus_queue.cc:260] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.514504 1231 raft_consensus.cc:3054] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:27.516242 1156 raft_consensus.cc:2463] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 4.
I20250114 02:53:27.516876 1045 leader_election.cc:304] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:27.517354 1397 raft_consensus.cc:2798] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:53:27.517656 1397 raft_consensus.cc:695] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 4 LEADER]: Becoming Leader. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:27.518072 1231 raft_consensus.cc:2463] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 4.
I20250114 02:53:27.518275 1397 consensus_queue.cc:237] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.523597 979 catalog_manager.cc:5526] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 reported cstate change: term changed from 3 to 4, leader changed from dba85f60dcc94b6797318aef42473236 (127.0.106.67) to fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65). New cstate: current_term: 4 leader_uuid: "fe3889e52a2c4bc2873b387533c90d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:27.644491 1231 raft_consensus.cc:1270] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:53:27.644488 1156 raft_consensus.cc:1270] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:53:27.645807 1392 consensus_queue.cc:1035] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:53:27.645931 1397 consensus_queue.cc:1035] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:53:27.652776 1396 raft_consensus.cc:988] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc: : Instructing follower fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:27.653146 1396 raft_consensus.cc:1076] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Signalling peer fe3889e52a2c4bc2873b387533c90d34 to start an election
I20250114 02:53:27.655889 1080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
from {username='slave'} at 127.0.0.1:44688
I20250114 02:53:27.656299 1080 raft_consensus.cc:491] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:27.656569 1080 raft_consensus.cc:3054] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:27.656859 1230 raft_consensus.cc:1270] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:53:27.658074 1392 consensus_queue.cc:1035] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:53:27.662053 1080 raft_consensus.cc:513] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.664489 1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:27.665026 1157 raft_consensus.cc:3049] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:27.665343 1157 raft_consensus.cc:738] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Running, Role: LEADER
I20250114 02:53:27.665745 1229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89" candidate_uuid: "fe3889e52a2c4bc2873b387533c90d34" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:27.666282 1229 raft_consensus.cc:3054] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:27.666137 1157 consensus_queue.cc:260] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.667347 1157 raft_consensus.cc:3054] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:27.672878 1229 raft_consensus.cc:2463] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 3.
I20250114 02:53:27.673494 1157 raft_consensus.cc:2463] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3889e52a2c4bc2873b387533c90d34 in term 3.
I20250114 02:53:27.673923 1047 leader_election.cc:304] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba85f60dcc94b6797318aef42473236, fe3889e52a2c4bc2873b387533c90d34; no voters:
I20250114 02:53:27.683202 1392 raft_consensus.cc:2798] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:27.683545 1392 raft_consensus.cc:695] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 3 LEADER]: Becoming Leader. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:27.684181 1392 consensus_queue.cc:237] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:27.688674 1080 leader_election.cc:290] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [CANDIDATE]: Term 3 election: Requested vote from peers 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:27.692836 1157 raft_consensus.cc:1270] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:53:27.695320 1414 consensus_queue.cc:1035] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 02:53:27.706918 979 catalog_manager.cc:5526] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 reported cstate change: term changed from 2 to 3, leader changed from 678704d2559e41eb9d8b76af869bbefc (127.0.106.66) to fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65). New cstate: current_term: 3 leader_uuid: "fe3889e52a2c4bc2873b387533c90d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:27.947461 1157 raft_consensus.cc:1270] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:53:27.948448 1413 consensus_queue.cc:1035] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:53:27.956715 1229 raft_consensus.cc:1270] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:53:27.957888 1413 consensus_queue.cc:1035] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:53:28.041045 1229 raft_consensus.cc:1270] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:53:28.041982 1411 consensus_queue.cc:1035] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:53:28.050894 1157 raft_consensus.cc:1270] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:53:28.053084 1411 consensus_queue.cc:1035] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:53:28.074397 1230 raft_consensus.cc:1270] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:53:28.075259 1411 consensus_queue.cc:1035] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:53:28.082660 1157 raft_consensus.cc:1270] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Refusing update from remote peer fe3889e52a2c4bc2873b387533c90d34: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:53:28.083863 1414 consensus_queue.cc:1035] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:53:30.224773 425 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20250114 02:53:30.225355 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:30.231660 1080 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ccf3b62f4d44f9d85522e68a32a800f"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
mode: GRACEFUL
new_leader_uuid: "678704d2559e41eb9d8b76af869bbefc"
from {username='slave'} at 127.0.0.1:55026
I20250114 02:53:30.232064 1080 raft_consensus.cc:604] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Received request to transfer leadership to TS 678704d2559e41eb9d8b76af869bbefc
I20250114 02:53:30.234104 1080 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "559ae9b489104111a990c10573de9259"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
mode: GRACEFUL
new_leader_uuid: "dba85f60dcc94b6797318aef42473236"
from {username='slave'} at 127.0.0.1:55026
I20250114 02:53:30.234490 1080 raft_consensus.cc:604] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Received request to transfer leadership to TS dba85f60dcc94b6797318aef42473236
I20250114 02:53:30.236114 1080 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
mode: GRACEFUL
new_leader_uuid: "dba85f60dcc94b6797318aef42473236"
from {username='slave'} at 127.0.0.1:55026
I20250114 02:53:30.236493 1080 raft_consensus.cc:604] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 3 LEADER]: Received request to transfer leadership to TS dba85f60dcc94b6797318aef42473236
I20250114 02:53:30.238302 1080 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0"
dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
mode: GRACEFUL
new_leader_uuid: "678704d2559e41eb9d8b76af869bbefc"
from {username='slave'} at 127.0.0.1:55026
I20250114 02:53:30.238806 1080 raft_consensus.cc:604] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 3 LEADER]: Received request to transfer leadership to TS 678704d2559e41eb9d8b76af869bbefc
I20250114 02:53:30.239548 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250114 02:53:30.239920 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:30.447141 1424 raft_consensus.cc:988] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34: : Instructing follower dba85f60dcc94b6797318aef42473236 to start an election
I20250114 02:53:30.447556 1411 raft_consensus.cc:1076] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 3 LEADER]: Signalling peer dba85f60dcc94b6797318aef42473236 to start an election
I20250114 02:53:30.448580 1230 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89"
dest_uuid: "dba85f60dcc94b6797318aef42473236"
from {username='slave'} at 127.0.0.1:37588
I20250114 02:53:30.448956 1230 raft_consensus.cc:491] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:30.449187 1230 raft_consensus.cc:3054] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:30.454214 1230 raft_consensus.cc:513] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:30.455811 1230 leader_election.cc:290] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 4 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:30.456424 1080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:30.456630 1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8452bf971d30409e81d5d997e366ed89" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:30.457048 1080 raft_consensus.cc:3049] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:53:30.457132 1157 raft_consensus.cc:3054] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:30.457440 1080 raft_consensus.cc:738] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:30.457980 1080 consensus_queue.cc:260] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:30.458722 1080 raft_consensus.cc:3054] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:30.460937 1157 raft_consensus.cc:2463] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 4.
I20250114 02:53:30.461730 1194 leader_election.cc:304] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters:
I20250114 02:53:30.462307 1080 raft_consensus.cc:2463] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 4.
I20250114 02:53:30.462334 1368 raft_consensus.cc:2798] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:53:30.463515 1368 raft_consensus.cc:695] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 4 LEADER]: Becoming Leader. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:30.464272 1368 consensus_queue.cc:237] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:30.469996 979 catalog_manager.cc:5526] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 reported cstate change: term changed from 3 to 4, leader changed from fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65) to dba85f60dcc94b6797318aef42473236 (127.0.106.67). New cstate: current_term: 4 leader_uuid: "dba85f60dcc94b6797318aef42473236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:30.643967 1411 raft_consensus.cc:988] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34: : Instructing follower dba85f60dcc94b6797318aef42473236 to start an election
I20250114 02:53:30.644434 1424 raft_consensus.cc:1076] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Signalling peer dba85f60dcc94b6797318aef42473236 to start an election
I20250114 02:53:30.645568 1230 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "559ae9b489104111a990c10573de9259"
dest_uuid: "dba85f60dcc94b6797318aef42473236"
from {username='slave'} at 127.0.0.1:37588
I20250114 02:53:30.645936 1230 raft_consensus.cc:491] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:30.646167 1230 raft_consensus.cc:3054] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:30.649827 1230 raft_consensus.cc:513] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:30.650967 1230 leader_election.cc:290] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 2 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721)
I20250114 02:53:30.651592 1080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ae9b489104111a990c10573de9259" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:30.651727 1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ae9b489104111a990c10573de9259" candidate_uuid: "dba85f60dcc94b6797318aef42473236" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
I20250114 02:53:30.652115 1080 raft_consensus.cc:3049] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:30.652205 1157 raft_consensus.cc:3054] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:30.652383 1080 raft_consensus.cc:738] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:30.652913 1080 consensus_queue.cc:260] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:30.653652 1080 raft_consensus.cc:3054] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:30.655592 1157 raft_consensus.cc:2463] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 2.
I20250114 02:53:30.656342 1194 leader_election.cc:304] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters:
I20250114 02:53:30.657022 1368 raft_consensus.cc:2798] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:30.657334 1368 raft_consensus.cc:695] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [term 2 LEADER]: Becoming Leader. State: Replica: dba85f60dcc94b6797318aef42473236, State: Running, Role: LEADER
I20250114 02:53:30.657452 1080 raft_consensus.cc:2463] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba85f60dcc94b6797318aef42473236 in term 2.
I20250114 02:53:30.657984 1368 consensus_queue.cc:237] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:30.664026 979 catalog_manager.cc:5526] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 reported cstate change: term changed from 1 to 2, leader changed from fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65) to dba85f60dcc94b6797318aef42473236 (127.0.106.67). New cstate: current_term: 2 leader_uuid: "dba85f60dcc94b6797318aef42473236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:30.707480 1411 raft_consensus.cc:988] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34: : Instructing follower 678704d2559e41eb9d8b76af869bbefc to start an election
I20250114 02:53:30.707834 1424 raft_consensus.cc:1076] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 3 LEADER]: Signalling peer 678704d2559e41eb9d8b76af869bbefc to start an election
I20250114 02:53:30.708803 1157 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0"
dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
from {username='slave'} at 127.0.0.1:32890
I20250114 02:53:30.709221 1157 raft_consensus.cc:491] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:30.709483 1157 raft_consensus.cc:3054] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:30.713085 1157 raft_consensus.cc:513] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:30.714632 1157 leader_election.cc:290] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 4 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:30.715124 1080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:30.715602 1080 raft_consensus.cc:3049] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:53:30.715626 1230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1518c696914d80b37027e1b8d4fda0" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:30.715939 1080 raft_consensus.cc:738] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:30.716133 1230 raft_consensus.cc:3054] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:30.716588 1080 consensus_queue.cc:260] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:30.717384 1080 raft_consensus.cc:3054] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:53:30.720695 1230 raft_consensus.cc:2463] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 4.
I20250114 02:53:30.721542 1122 leader_election.cc:304] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters:
I20250114 02:53:30.721609 1080 raft_consensus.cc:2463] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 4.
I20250114 02:53:30.722151 1396 raft_consensus.cc:2798] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:53:30.723079 1396 raft_consensus.cc:695] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 4 LEADER]: Becoming Leader. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Running, Role: LEADER
I20250114 02:53:30.723667 1396 consensus_queue.cc:237] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:30.729334 979 catalog_manager.cc:5526] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc reported cstate change: term changed from 3 to 4, leader changed from fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65) to 678704d2559e41eb9d8b76af869bbefc (127.0.106.66). New cstate: current_term: 4 leader_uuid: "678704d2559e41eb9d8b76af869bbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:30.734596 1411 raft_consensus.cc:988] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34: : Instructing follower 678704d2559e41eb9d8b76af869bbefc to start an election
I20250114 02:53:30.734977 1424 raft_consensus.cc:1076] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Signalling peer 678704d2559e41eb9d8b76af869bbefc to start an election
I20250114 02:53:30.735939 1157 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ccf3b62f4d44f9d85522e68a32a800f"
dest_uuid: "678704d2559e41eb9d8b76af869bbefc"
from {username='slave'} at 127.0.0.1:32890
I20250114 02:53:30.736251 1157 raft_consensus.cc:491] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 02:53:30.736438 1157 raft_consensus.cc:3054] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:30.739781 1157 raft_consensus.cc:513] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:30.740904 1157 leader_election.cc:290] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 2 election: Requested vote from peers fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803), dba85f60dcc94b6797318aef42473236 (127.0.106.67:34357)
I20250114 02:53:30.741436 1080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccf3b62f4d44f9d85522e68a32a800f" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3889e52a2c4bc2873b387533c90d34"
I20250114 02:53:30.741844 1080 raft_consensus.cc:3049] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:30.741788 1230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccf3b62f4d44f9d85522e68a32a800f" candidate_uuid: "678704d2559e41eb9d8b76af869bbefc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dba85f60dcc94b6797318aef42473236"
I20250114 02:53:30.742100 1080 raft_consensus.cc:738] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3889e52a2c4bc2873b387533c90d34, State: Running, Role: LEADER
I20250114 02:53:30.742274 1230 raft_consensus.cc:3054] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:30.742686 1080 consensus_queue.cc:260] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:30.743475 1080 raft_consensus.cc:3054] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:30.745862 1230 raft_consensus.cc:2463] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 2.
I20250114 02:53:30.746482 1122 leader_election.cc:304] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678704d2559e41eb9d8b76af869bbefc, dba85f60dcc94b6797318aef42473236; no voters:
I20250114 02:53:30.747022 1396 raft_consensus.cc:2798] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:30.747072 1080 raft_consensus.cc:2463] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678704d2559e41eb9d8b76af869bbefc in term 2.
I20250114 02:53:30.747382 1396 raft_consensus.cc:695] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Becoming Leader. State: Replica: 678704d2559e41eb9d8b76af869bbefc, State: Running, Role: LEADER
I20250114 02:53:30.748145 1396 consensus_queue.cc:237] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } }
I20250114 02:53:30.753105 979 catalog_manager.cc:5526] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc reported cstate change: term changed from 1 to 2, leader changed from fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65) to 678704d2559e41eb9d8b76af869bbefc (127.0.106.66). New cstate: current_term: 2 leader_uuid: "678704d2559e41eb9d8b76af869bbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:30.916352 1157 raft_consensus.cc:1270] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Refusing update from remote peer dba85f60dcc94b6797318aef42473236: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:53:30.917249 1368 consensus_queue.cc:1035] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:53:30.925359 1080 raft_consensus.cc:1270] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Refusing update from remote peer dba85f60dcc94b6797318aef42473236: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:53:30.926327 1368 consensus_queue.cc:1035] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:53:31.130879 1230 raft_consensus.cc:1270] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Refusing update from remote peer 678704d2559e41eb9d8b76af869bbefc: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:53:31.131765 1396 consensus_queue.cc:1035] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:53:31.138496 1080 raft_consensus.cc:1270] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Refusing update from remote peer 678704d2559e41eb9d8b76af869bbefc: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:53:31.138957 1157 raft_consensus.cc:1270] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Refusing update from remote peer dba85f60dcc94b6797318aef42473236: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:31.140089 1432 consensus_queue.cc:1035] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:53:31.140938 1368 consensus_queue.cc:1035] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678704d2559e41eb9d8b76af869bbefc" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:31.152078 1080 raft_consensus.cc:1270] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Refusing update from remote peer dba85f60dcc94b6797318aef42473236: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:31.153252 1368 consensus_queue.cc:1035] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:31.212404 1230 raft_consensus.cc:1270] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Refusing update from remote peer 678704d2559e41eb9d8b76af869bbefc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:31.213896 1432 consensus_queue.cc:1035] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba85f60dcc94b6797318aef42473236" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:53:31.226003 1080 raft_consensus.cc:1270] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Refusing update from remote peer 678704d2559e41eb9d8b76af869bbefc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:31.228011 1441 consensus_queue.cc:1035] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3889e52a2c4bc2873b387533c90d34" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:53:31.240721 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:31.247493 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:31.247824 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:32.248591 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:32.253386 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:32.253782 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:33.254487 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:33.260354 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:33.260675 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:34.261296 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:34.266016 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:34.266384 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:35.266989 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:35.271751 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:35.272117 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:36.272776 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:36.277230 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:36.277544 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:37.278169 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:37.283955 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:37.284360 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:38.285027 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:38.289991 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:38.290364 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:39.291024 425 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250114 02:53:39.295717 425 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250114 02:53:39.296137 425 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250114 02:53:40.319424 425 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20250114 02:53:40.319959 425 tablet_server.cc:178] TabletServer@127.0.106.65:0 shutting down...
I20250114 02:53:40.345278 425 ts_tablet_manager.cc:1500] Shutting down tablet manager...
W20250114 02:53:40.345602 1195 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
I20250114 02:53:40.346127 425 tablet_replica.cc:331] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34: stopping tablet replica
I20250114 02:53:40.346778 425 raft_consensus.cc:2238] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 4 LEADER]: Raft consensus shutting down.
I20250114 02:53:40.347635 425 raft_consensus.cc:2267] T 7b3949440ca34a658306d0065161a875 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Raft consensus is shut down!
W20250114 02:53:40.349493 1195 consensus_peers.cc:487] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 -> Peer fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803): Couldn't send request to peer fe3889e52a2c4bc2873b387533c90d34. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:40.349828 425 tablet_replica.cc:331] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34: stopping tablet replica
I20250114 02:53:40.350356 425 raft_consensus.cc:2238] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.350764 425 raft_consensus.cc:2267] T db1518c696914d80b37027e1b8d4fda0 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.352403 425 tablet_replica.cc:331] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34: stopping tablet replica
I20250114 02:53:40.352846 425 raft_consensus.cc:2238] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 3 LEADER]: Raft consensus shutting down.
I20250114 02:53:40.353441 425 raft_consensus.cc:2267] T bfb6d62d39fa47059bcf3d88f235c999 P fe3889e52a2c4bc2873b387533c90d34 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.354779 425 tablet_replica.cc:331] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34: stopping tablet replica
I20250114 02:53:40.355209 425 raft_consensus.cc:2238] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 4 LEADER]: Raft consensus shutting down.
I20250114 02:53:40.355844 425 raft_consensus.cc:2267] T bdc1d29533f046f9997fc1dd89c84722 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.357347 425 tablet_replica.cc:331] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34: stopping tablet replica
I20250114 02:53:40.357774 425 raft_consensus.cc:2238] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.358157 425 raft_consensus.cc:2267] T 8452bf971d30409e81d5d997e366ed89 P fe3889e52a2c4bc2873b387533c90d34 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.359683 425 tablet_replica.cc:331] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34: stopping tablet replica
I20250114 02:53:40.360123 425 raft_consensus.cc:2238] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:53:40.360841 425 raft_consensus.cc:2267] T 3144d42016f44d539f4c03185a17cd64 P fe3889e52a2c4bc2873b387533c90d34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.362329 425 tablet_replica.cc:331] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34: stopping tablet replica
I20250114 02:53:40.362793 425 raft_consensus.cc:2238] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.363150 425 raft_consensus.cc:2267] T 7415008b1fac4680894ed783f47e65fd P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.364758 425 tablet_replica.cc:331] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34: stopping tablet replica
I20250114 02:53:40.365150 425 raft_consensus.cc:2238] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.365468 425 raft_consensus.cc:2267] T 3ccf3b62f4d44f9d85522e68a32a800f P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.366799 425 tablet_replica.cc:331] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34: stopping tablet replica
I20250114 02:53:40.367241 425 raft_consensus.cc:2238] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.367616 425 raft_consensus.cc:2267] T bebf4d4a188940c685b6726024522268 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.369128 425 tablet_replica.cc:331] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34: stopping tablet replica
I20250114 02:53:40.369535 425 raft_consensus.cc:2238] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.369918 425 raft_consensus.cc:2267] T 559ae9b489104111a990c10573de9259 P fe3889e52a2c4bc2873b387533c90d34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.397313 425 tablet_server.cc:195] TabletServer@127.0.106.65:0 shutdown complete.
W20250114 02:53:40.408123 1121 consensus_peers.cc:487] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc -> Peer fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803): Couldn't send request to peer fe3889e52a2c4bc2873b387533c90d34. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.65:42803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:40.417918 425 tablet_server.cc:178] TabletServer@127.0.106.66:0 shutting down...
I20250114 02:53:40.445001 425 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 02:53:40.445804 425 tablet_replica.cc:331] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc: stopping tablet replica
W20250114 02:53:40.446497 1194 consensus_peers.cc:487] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 -> Peer 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721): Couldn't send request to peer 678704d2559e41eb9d8b76af869bbefc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:40.446877 425 raft_consensus.cc:2238] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.447368 425 raft_consensus.cc:2267] T 3144d42016f44d539f4c03185a17cd64 P 678704d2559e41eb9d8b76af869bbefc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.448966 425 tablet_replica.cc:331] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc: stopping tablet replica
I20250114 02:53:40.449427 425 raft_consensus.cc:2238] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.449816 425 raft_consensus.cc:2267] T 7b3949440ca34a658306d0065161a875 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.451253 425 tablet_replica.cc:331] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc: stopping tablet replica
I20250114 02:53:40.451684 425 raft_consensus.cc:2238] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 4 LEADER]: Raft consensus shutting down.
I20250114 02:53:40.452337 425 raft_consensus.cc:2267] T db1518c696914d80b37027e1b8d4fda0 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.453809 425 tablet_replica.cc:331] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc: stopping tablet replica
I20250114 02:53:40.454210 425 raft_consensus.cc:2238] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.454546 425 raft_consensus.cc:2267] T bfb6d62d39fa47059bcf3d88f235c999 P 678704d2559e41eb9d8b76af869bbefc [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.456216 425 tablet_replica.cc:331] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc: stopping tablet replica
I20250114 02:53:40.456642 425 raft_consensus.cc:2238] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.456971 425 raft_consensus.cc:2267] T bdc1d29533f046f9997fc1dd89c84722 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.458477 425 tablet_replica.cc:331] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc: stopping tablet replica
I20250114 02:53:40.458920 425 raft_consensus.cc:2238] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.459300 425 raft_consensus.cc:2267] T 8452bf971d30409e81d5d997e366ed89 P 678704d2559e41eb9d8b76af869bbefc [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.460744 425 tablet_replica.cc:331] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc: stopping tablet replica
I20250114 02:53:40.461119 425 raft_consensus.cc:2238] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.461416 425 raft_consensus.cc:2267] T 559ae9b489104111a990c10573de9259 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.462801 425 tablet_replica.cc:331] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc: stopping tablet replica
I20250114 02:53:40.463204 425 raft_consensus.cc:2238] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Raft consensus shutting down.
I20250114 02:53:40.463932 425 raft_consensus.cc:2267] T bebf4d4a188940c685b6726024522268 P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.465443 425 tablet_replica.cc:331] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc: stopping tablet replica
I20250114 02:53:40.465849 425 raft_consensus.cc:2238] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.466193 425 raft_consensus.cc:2267] T 7415008b1fac4680894ed783f47e65fd P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.467708 425 tablet_replica.cc:331] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc: stopping tablet replica
I20250114 02:53:40.468098 425 raft_consensus.cc:2238] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [term 2 LEADER]: Raft consensus shutting down.
I20250114 02:53:40.468775 425 raft_consensus.cc:2267] T 3ccf3b62f4d44f9d85522e68a32a800f P 678704d2559e41eb9d8b76af869bbefc [term 2 FOLLOWER]: Raft consensus is shut down!
W20250114 02:53:40.481604 1195 consensus_peers.cc:487] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 -> Peer fe3889e52a2c4bc2873b387533c90d34 (127.0.106.65:42803): Couldn't send request to peer fe3889e52a2c4bc2873b387533c90d34. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.65:42803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:40.495832 425 tablet_server.cc:195] TabletServer@127.0.106.66:0 shutdown complete.
I20250114 02:53:40.514089 425 tablet_server.cc:178] TabletServer@127.0.106.67:0 shutting down...
W20250114 02:53:40.525027 1194 consensus_peers.cc:487] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 -> Peer 678704d2559e41eb9d8b76af869bbefc (127.0.106.66:34721): Couldn't send request to peer 678704d2559e41eb9d8b76af869bbefc. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.66:34721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:40.537569 425 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 02:53:40.538239 425 tablet_replica.cc:331] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236: stopping tablet replica
I20250114 02:53:40.539065 425 raft_consensus.cc:2238] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.539494 425 raft_consensus.cc:2267] T db1518c696914d80b37027e1b8d4fda0 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.541164 425 tablet_replica.cc:331] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236: stopping tablet replica
I20250114 02:53:40.541628 425 raft_consensus.cc:2238] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 4 LEADER]: Raft consensus shutting down.
I20250114 02:53:40.542322 425 raft_consensus.cc:2267] T 8452bf971d30409e81d5d997e366ed89 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.543890 425 tablet_replica.cc:331] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236: stopping tablet replica
I20250114 02:53:40.544338 425 raft_consensus.cc:2238] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.544716 425 raft_consensus.cc:2267] T bfb6d62d39fa47059bcf3d88f235c999 P dba85f60dcc94b6797318aef42473236 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.546187 425 tablet_replica.cc:331] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236: stopping tablet replica
I20250114 02:53:40.546658 425 raft_consensus.cc:2238] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.547029 425 raft_consensus.cc:2267] T bdc1d29533f046f9997fc1dd89c84722 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.548524 425 tablet_replica.cc:331] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236: stopping tablet replica
I20250114 02:53:40.548969 425 raft_consensus.cc:2238] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.549320 425 raft_consensus.cc:2267] T 3144d42016f44d539f4c03185a17cd64 P dba85f60dcc94b6797318aef42473236 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.550827 425 tablet_replica.cc:331] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236: stopping tablet replica
I20250114 02:53:40.551283 425 raft_consensus.cc:2238] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.551664 425 raft_consensus.cc:2267] T 7b3949440ca34a658306d0065161a875 P dba85f60dcc94b6797318aef42473236 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.553150 425 tablet_replica.cc:331] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236: stopping tablet replica
I20250114 02:53:40.553570 425 raft_consensus.cc:2238] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.553943 425 raft_consensus.cc:2267] T bebf4d4a188940c685b6726024522268 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.555315 425 tablet_replica.cc:331] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236: stopping tablet replica
I20250114 02:53:40.555728 425 raft_consensus.cc:2238] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [term 2 LEADER]: Raft consensus shutting down.
I20250114 02:53:40.556362 425 raft_consensus.cc:2267] T 559ae9b489104111a990c10573de9259 P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.557698 425 tablet_replica.cc:331] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236: stopping tablet replica
I20250114 02:53:40.558084 425 raft_consensus.cc:2238] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [term 2 LEADER]: Raft consensus shutting down.
I20250114 02:53:40.558727 425 raft_consensus.cc:2267] T 7415008b1fac4680894ed783f47e65fd P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.560171 425 tablet_replica.cc:331] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236: stopping tablet replica
I20250114 02:53:40.560564 425 raft_consensus.cc:2238] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 02:53:40.560876 425 raft_consensus.cc:2267] T 3ccf3b62f4d44f9d85522e68a32a800f P dba85f60dcc94b6797318aef42473236 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.600868 425 tablet_server.cc:195] TabletServer@127.0.106.67:0 shutdown complete.
I20250114 02:53:40.617964 425 master.cc:537] Master@127.0.106.126:38423 shutting down...
I20250114 02:53:40.637964 425 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:53:40.638464 425 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:53:40.638793 425 tablet_replica.cc:331] T 00000000000000000000000000000000 P 3d0f9c5915b6496bb155c7526d0128a9: stopping tablet replica
I20250114 02:53:40.657860 425 master.cc:559] Master@127.0.106.126:38423 shutdown complete.
678704d2559e41eb9d8b76af869bbefc 4
dba85f60dcc94b6797318aef42473236 1
fe3889e52a2c4bc2873b387533c90d34 5
678704d2559e41eb9d8b76af869bbefc 4
dba85f60dcc94b6797318aef42473236 3
fe3889e52a2c4bc2873b387533c90d34 3
678704d2559e41eb9d8b76af869bbefc 1
dba85f60dcc94b6797318aef42473236 1
fe3889e52a2c4bc2873b387533c90d34 8
678704d2559e41eb9d8b76af869bbefc 3
dba85f60dcc94b6797318aef42473236 3
fe3889e52a2c4bc2873b387533c90d34 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28621 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20250114 02:53:40.696811 425 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.106.126:37383
I20250114 02:53:40.697813 425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:53:40.703999 1460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:40.705055 1461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:40.705986 1463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:40.707060 425 server_base.cc:1034] running on GCE node
I20250114 02:53:40.707939 425 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 02:53:40.708101 425 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 02:53:40.708225 425 hybrid_clock.cc:648] HybridClock initialized: now 1736823220708209 us; error 0 us; skew 500 ppm
I20250114 02:53:40.708680 425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:53:40.711095 425 webserver.cc:458] Webserver started at http://127.0.106.126:33027/ using document root <none> and password file <none>
I20250114 02:53:40.711539 425 fs_manager.cc:362] Metadata directory not provided
I20250114 02:53:40.711688 425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:53:40.711925 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:53:40.713024 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/master-0-root/instance:
uuid: "e04023df409b4be7b6eb175399d2fd72"
format_stamp: "Formatted at 2025-01-14 02:53:40 on dist-test-slave-k78q"
I20250114 02:53:40.716979 425 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 02:53:40.719923 1468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:40.720577 425 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 02:53:40.720813 425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/master-0-root
uuid: "e04023df409b4be7b6eb175399d2fd72"
format_stamp: "Formatted at 2025-01-14 02:53:40 on dist-test-slave-k78q"
I20250114 02:53:40.721053 425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:53:40.733915 425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:53:40.734920 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:53:40.767741 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.126:37383
I20250114 02:53:40.767825 1519 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.126:37383 every 8 connection(s)
I20250114 02:53:40.771284 1520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:40.779800 1520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72: Bootstrap starting.
I20250114 02:53:40.783903 1520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:40.787688 1520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72: No bootstrap required, opened a new log
I20250114 02:53:40.789538 1520 raft_consensus.cc:357] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04023df409b4be7b6eb175399d2fd72" member_type: VOTER }
I20250114 02:53:40.789927 1520 raft_consensus.cc:383] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:40.790140 1520 raft_consensus.cc:738] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04023df409b4be7b6eb175399d2fd72, State: Initialized, Role: FOLLOWER
I20250114 02:53:40.790693 1520 consensus_queue.cc:260] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04023df409b4be7b6eb175399d2fd72" member_type: VOTER }
I20250114 02:53:40.791110 1520 raft_consensus.cc:397] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 02:53:40.791298 1520 raft_consensus.cc:491] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 02:53:40.791538 1520 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:40.795118 1520 raft_consensus.cc:513] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04023df409b4be7b6eb175399d2fd72" member_type: VOTER }
I20250114 02:53:40.795557 1520 leader_election.cc:304] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e04023df409b4be7b6eb175399d2fd72; no voters:
I20250114 02:53:40.796478 1520 leader_election.cc:290] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 02:53:40.796840 1523 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:40.797961 1523 raft_consensus.cc:695] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [term 1 LEADER]: Becoming Leader. State: Replica: e04023df409b4be7b6eb175399d2fd72, State: Running, Role: LEADER
I20250114 02:53:40.798624 1523 consensus_queue.cc:237] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04023df409b4be7b6eb175399d2fd72" member_type: VOTER }
I20250114 02:53:40.799221 1520 sys_catalog.cc:564] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 02:53:40.802641 1524 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e04023df409b4be7b6eb175399d2fd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04023df409b4be7b6eb175399d2fd72" member_type: VOTER } }
I20250114 02:53:40.803377 1524 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [sys.catalog]: This master's current role is: LEADER
I20250114 02:53:40.803881 1525 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e04023df409b4be7b6eb175399d2fd72. Latest consensus state: current_term: 1 leader_uuid: "e04023df409b4be7b6eb175399d2fd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04023df409b4be7b6eb175399d2fd72" member_type: VOTER } }
I20250114 02:53:40.804605 1525 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72 [sys.catalog]: This master's current role is: LEADER
I20250114 02:53:40.805060 1530 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 02:53:40.810205 1530 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 02:53:40.814266 425 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 02:53:40.818113 1530 catalog_manager.cc:1348] Generated new cluster ID: 5ac4fededa4b4fabb049190e8ee6de06
I20250114 02:53:40.818373 1530 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 02:53:40.835664 1530 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 02:53:40.836724 1530 catalog_manager.cc:1505] Loading token signing keys...
I20250114 02:53:40.855335 1530 catalog_manager.cc:5899] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72: Generated new TSK 0
I20250114 02:53:40.855871 1530 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 02:53:40.880359 425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:53:40.886723 1541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:40.887634 1542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:40.888784 1544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:40.889794 425 server_base.cc:1034] running on GCE node
I20250114 02:53:40.890642 425 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 02:53:40.890811 425 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 02:53:40.891000 425 hybrid_clock.cc:648] HybridClock initialized: now 1736823220890983 us; error 0 us; skew 500 ppm
I20250114 02:53:40.891520 425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:53:40.893685 425 webserver.cc:458] Webserver started at http://127.0.106.65:41921/ using document root <none> and password file <none>
I20250114 02:53:40.894114 425 fs_manager.cc:362] Metadata directory not provided
I20250114 02:53:40.894282 425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:53:40.894515 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:53:40.895622 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-0-root/instance:
uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
format_stamp: "Formatted at 2025-01-14 02:53:40 on dist-test-slave-k78q"
I20250114 02:53:40.899727 425 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250114 02:53:40.902695 1549 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:40.903331 425 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 02:53:40.903594 425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-0-root
uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
format_stamp: "Formatted at 2025-01-14 02:53:40 on dist-test-slave-k78q"
I20250114 02:53:40.903848 425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:53:40.926345 425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:53:40.927558 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:53:40.928834 425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:53:40.931025 425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:53:40.931207 425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:40.931440 425 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:53:40.931583 425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:40.967418 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.65:46625
I20250114 02:53:40.967505 1611 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.65:46625 every 8 connection(s)
I20250114 02:53:40.971638 425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:53:40.979562 1616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:40.980755 1617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:40.982753 1619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:40.983718 425 server_base.cc:1034] running on GCE node
I20250114 02:53:40.983912 1612 heartbeater.cc:346] Connected to a master server at 127.0.106.126:37383
I20250114 02:53:40.984277 1612 heartbeater.cc:463] Registering TS with master...
I20250114 02:53:40.985124 1612 heartbeater.cc:510] Master 127.0.106.126:37383 requested a full tablet report, sending...
I20250114 02:53:40.985477 425 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 02:53:40.985706 425 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 02:53:40.985893 425 hybrid_clock.cc:648] HybridClock initialized: now 1736823220985868 us; error 0 us; skew 500 ppm
I20250114 02:53:40.986529 425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:53:40.986956 1485 ts_manager.cc:194] Registered new tserver with Master: 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625)
I20250114 02:53:40.988837 1485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39724
I20250114 02:53:40.989646 425 webserver.cc:458] Webserver started at http://127.0.106.66:37591/ using document root <none> and password file <none>
I20250114 02:53:40.990247 425 fs_manager.cc:362] Metadata directory not provided
I20250114 02:53:40.990510 425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:53:40.990832 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:53:40.991988 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-1-root/instance:
uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
format_stamp: "Formatted at 2025-01-14 02:53:40 on dist-test-slave-k78q"
I20250114 02:53:40.995739 425 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 02:53:40.998394 1624 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:40.999118 425 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 02:53:40.999346 425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-1-root
uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
format_stamp: "Formatted at 2025-01-14 02:53:40 on dist-test-slave-k78q"
I20250114 02:53:40.999577 425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:53:41.015640 425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:53:41.016688 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:53:41.017958 425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:53:41.020054 425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:53:41.020232 425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:41.020463 425 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:53:41.020593 425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:41.056470 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.66:37511
I20250114 02:53:41.056573 1686 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.66:37511 every 8 connection(s)
I20250114 02:53:41.060599 425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:53:41.068325 1690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:41.069351 1691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:41.070477 425 server_base.cc:1034] running on GCE node
W20250114 02:53:41.071283 1693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:41.072152 425 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250114 02:53:41.072292 1687 heartbeater.cc:346] Connected to a master server at 127.0.106.126:37383
W20250114 02:53:41.072439 425 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 02:53:41.072687 1687 heartbeater.cc:463] Registering TS with master...
I20250114 02:53:41.072660 425 hybrid_clock.cc:648] HybridClock initialized: now 1736823221072643 us; error 0 us; skew 500 ppm
I20250114 02:53:41.073478 425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:53:41.073580 1687 heartbeater.cc:510] Master 127.0.106.126:37383 requested a full tablet report, sending...
I20250114 02:53:41.075558 1485 ts_manager.cc:194] Registered new tserver with Master: 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:41.076145 425 webserver.cc:458] Webserver started at http://127.0.106.67:45897/ using document root <none> and password file <none>
I20250114 02:53:41.076617 425 fs_manager.cc:362] Metadata directory not provided
I20250114 02:53:41.076802 425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:53:41.076859 1485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39732
I20250114 02:53:41.077047 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:53:41.078446 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-2-root/instance:
uuid: "b89550620e1d455d887be72232bb5454"
format_stamp: "Formatted at 2025-01-14 02:53:41 on dist-test-slave-k78q"
I20250114 02:53:41.082439 425 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250114 02:53:41.085335 1698 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:41.085975 425 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250114 02:53:41.086210 425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-2-root
uuid: "b89550620e1d455d887be72232bb5454"
format_stamp: "Formatted at 2025-01-14 02:53:41 on dist-test-slave-k78q"
I20250114 02:53:41.086445 425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:53:41.093767 425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:53:41.094826 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:53:41.095995 425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:53:41.097862 425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:53:41.098030 425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:41.098230 425 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:53:41.098378 425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:41.134639 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.67:42283
I20250114 02:53:41.134735 1760 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.67:42283 every 8 connection(s)
I20250114 02:53:41.147082 1761 heartbeater.cc:346] Connected to a master server at 127.0.106.126:37383
I20250114 02:53:41.147437 1761 heartbeater.cc:463] Registering TS with master...
I20250114 02:53:41.148064 1761 heartbeater.cc:510] Master 127.0.106.126:37383 requested a full tablet report, sending...
I20250114 02:53:41.149708 1485 ts_manager.cc:194] Registered new tserver with Master: b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:41.150653 425 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013020679s
I20250114 02:53:41.150974 1485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39748
I20250114 02:53:41.153082 425 test_util.cc:274] Using random seed: -1423807197
I20250114 02:53:41.173934 1485 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39758:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:53:41.179813 1485 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:53:41.406167 1577 tablet_service.cc:1467] Processing CreateTablet for tablet 29b78b6bf8a94ca2b4d5f71aaad16e83 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250114 02:53:41.406919 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 29b78b6bf8a94ca2b4d5f71aaad16e83 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250114 02:53:41.408200 1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29b78b6bf8a94ca2b4d5f71aaad16e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.408269 1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29b78b6bf8a94ca2b4d5f71aaad16e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.414290 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4d46461bf745639ea4f64fa3bcb996 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250114 02:53:41.415570 1651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d4d46461bf745639ea4f64fa3bcb996. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.416352 1576 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4d46461bf745639ea4f64fa3bcb996 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250114 02:53:41.417636 1576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d4d46461bf745639ea4f64fa3bcb996. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.419425 1726 tablet_service.cc:1467] Processing CreateTablet for tablet 29b78b6bf8a94ca2b4d5f71aaad16e83 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250114 02:53:41.420867 1726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29b78b6bf8a94ca2b4d5f71aaad16e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.421658 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1c92618e4c44119a063bfdaa75fc3e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250114 02:53:41.422927 1646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a1c92618e4c44119a063bfdaa75fc3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.423383 1725 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4d46461bf745639ea4f64fa3bcb996 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250114 02:53:41.424161 1575 tablet_service.cc:1467] Processing CreateTablet for tablet ee328440ef8f409ca059ac947d40a58a (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250114 02:53:41.424636 1725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d4d46461bf745639ea4f64fa3bcb996. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.425374 1575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee328440ef8f409ca059ac947d40a58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.427790 1647 tablet_service.cc:1467] Processing CreateTablet for tablet a26544b19eeb4116a06caaba1da0e9d2 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250114 02:53:41.428992 1647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a26544b19eeb4116a06caaba1da0e9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.431799 1648 tablet_service.cc:1467] Processing CreateTablet for tablet f83a9dac23e1497286938e63aa482439 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250114 02:53:41.433107 1648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f83a9dac23e1497286938e63aa482439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.434644 1574 tablet_service.cc:1467] Processing CreateTablet for tablet b4aa8b25b08141db9f0576ce3fa34d3c (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250114 02:53:41.435739 1649 tablet_service.cc:1467] Processing CreateTablet for tablet b4aa8b25b08141db9f0576ce3fa34d3c (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250114 02:53:41.436291 1574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4aa8b25b08141db9f0576ce3fa34d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.436978 1649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4aa8b25b08141db9f0576ce3fa34d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.438375 1776 tablet_bootstrap.cc:492] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:41.439790 1650 tablet_service.cc:1467] Processing CreateTablet for tablet ee328440ef8f409ca059ac947d40a58a (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250114 02:53:41.441057 1650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee328440ef8f409ca059ac947d40a58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.441654 1573 tablet_service.cc:1467] Processing CreateTablet for tablet f83a9dac23e1497286938e63aa482439 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250114 02:53:41.442953 1573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f83a9dac23e1497286938e63aa482439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.443718 1724 tablet_service.cc:1467] Processing CreateTablet for tablet ee328440ef8f409ca059ac947d40a58a (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250114 02:53:41.444964 1724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee328440ef8f409ca059ac947d40a58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.445255 1776 tablet_bootstrap.cc:654] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:41.447752 1723 tablet_service.cc:1467] Processing CreateTablet for tablet b4aa8b25b08141db9f0576ce3fa34d3c (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250114 02:53:41.450043 1645 tablet_service.cc:1467] Processing CreateTablet for tablet c39189f5c30e4eec95701fe6a370896e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250114 02:53:41.450620 1723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4aa8b25b08141db9f0576ce3fa34d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.451428 1645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c39189f5c30e4eec95701fe6a370896e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.451889 1721 tablet_service.cc:1467] Processing CreateTablet for tablet a26544b19eeb4116a06caaba1da0e9d2 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250114 02:53:41.452972 1721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a26544b19eeb4116a06caaba1da0e9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.450043 1572 tablet_service.cc:1467] Processing CreateTablet for tablet a26544b19eeb4116a06caaba1da0e9d2 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250114 02:53:41.455634 1720 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1c92618e4c44119a063bfdaa75fc3e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250114 02:53:41.456916 1720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a1c92618e4c44119a063bfdaa75fc3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.450105 1722 tablet_service.cc:1467] Processing CreateTablet for tablet f83a9dac23e1497286938e63aa482439 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250114 02:53:41.457219 1571 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1c92618e4c44119a063bfdaa75fc3e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250114 02:53:41.457643 1570 tablet_service.cc:1467] Processing CreateTablet for tablet c39189f5c30e4eec95701fe6a370896e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250114 02:53:41.458235 1722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f83a9dac23e1497286938e63aa482439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.455650 1572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a26544b19eeb4116a06caaba1da0e9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.461071 1651 tablet_service.cc:1467] Processing CreateTablet for tablet be63bb7ee74c418aabd9237a6f6a0454 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250114 02:53:41.462184 1651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be63bb7ee74c418aabd9237a6f6a0454. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.462862 1570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c39189f5c30e4eec95701fe6a370896e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.462502 1569 tablet_service.cc:1467] Processing CreateTablet for tablet be63bb7ee74c418aabd9237a6f6a0454 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250114 02:53:41.463831 1569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be63bb7ee74c418aabd9237a6f6a0454. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.464315 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 460a8d6305b54a66ab67e039cfaaee26 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250114 02:53:41.465516 1644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 460a8d6305b54a66ab67e039cfaaee26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.467562 1568 tablet_service.cc:1467] Processing CreateTablet for tablet 460a8d6305b54a66ab67e039cfaaee26 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250114 02:53:41.468737 1568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 460a8d6305b54a66ab67e039cfaaee26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.471696 1643 tablet_service.cc:1467] Processing CreateTablet for tablet cd7d01e627ea4bfc86adf4c4fa3f6139 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250114 02:53:41.472043 1719 tablet_service.cc:1467] Processing CreateTablet for tablet c39189f5c30e4eec95701fe6a370896e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250114 02:53:41.473627 1719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c39189f5c30e4eec95701fe6a370896e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.481148 1571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a1c92618e4c44119a063bfdaa75fc3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.482870 1643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd7d01e627ea4bfc86adf4c4fa3f6139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.488595 1718 tablet_service.cc:1467] Processing CreateTablet for tablet be63bb7ee74c418aabd9237a6f6a0454 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250114 02:53:41.489836 1718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be63bb7ee74c418aabd9237a6f6a0454. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.496070 1717 tablet_service.cc:1467] Processing CreateTablet for tablet 460a8d6305b54a66ab67e039cfaaee26 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250114 02:53:41.497164 1717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 460a8d6305b54a66ab67e039cfaaee26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.513777 1720 tablet_service.cc:1467] Processing CreateTablet for tablet cd7d01e627ea4bfc86adf4c4fa3f6139 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250114 02:53:41.518481 1720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd7d01e627ea4bfc86adf4c4fa3f6139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.544046 1571 tablet_service.cc:1467] Processing CreateTablet for tablet cd7d01e627ea4bfc86adf4c4fa3f6139 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250114 02:53:41.545387 1571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd7d01e627ea4bfc86adf4c4fa3f6139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.555768 1652 tablet_service.cc:1467] Processing CreateTablet for tablet ef83c2f4c7494118a6564eaca6b7799d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250114 02:53:41.560961 1570 tablet_service.cc:1467] Processing CreateTablet for tablet fb007689a5d64ee2a8fa67adad56de19 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250114 02:53:41.563377 1570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb007689a5d64ee2a8fa67adad56de19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.562469 1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef83c2f4c7494118a6564eaca6b7799d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.562206 1648 tablet_service.cc:1467] Processing CreateTablet for tablet fb007689a5d64ee2a8fa67adad56de19 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250114 02:53:41.559358 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 98adfd690b3746baa16ad8c1b99c83bd (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250114 02:53:41.561408 1569 tablet_service.cc:1467] Processing CreateTablet for tablet 98adfd690b3746baa16ad8c1b99c83bd (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250114 02:53:41.563282 1725 tablet_service.cc:1467] Processing CreateTablet for tablet 98adfd690b3746baa16ad8c1b99c83bd (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250114 02:53:41.567098 1720 tablet_service.cc:1467] Processing CreateTablet for tablet ef83c2f4c7494118a6564eaca6b7799d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250114 02:53:41.583514 1569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98adfd690b3746baa16ad8c1b99c83bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.583185 1723 tablet_service.cc:1467] Processing CreateTablet for tablet edafd45551ec43f9a5b750c1e54c62f6 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250114 02:53:41.584619 1573 tablet_service.cc:1467] Processing CreateTablet for tablet ef83c2f4c7494118a6564eaca6b7799d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250114 02:53:41.585816 1573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef83c2f4c7494118a6564eaca6b7799d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.583536 1725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98adfd690b3746baa16ad8c1b99c83bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.590672 1723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edafd45551ec43f9a5b750c1e54c62f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.590781 1648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb007689a5d64ee2a8fa67adad56de19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.591813 1649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98adfd690b3746baa16ad8c1b99c83bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.597970 1722 tablet_service.cc:1467] Processing CreateTablet for tablet f8434d11346144778e385ac18d417859 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250114 02:53:41.600545 1776 tablet_bootstrap.cc:492] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:41.605859 1575 tablet_service.cc:1467] Processing CreateTablet for tablet f8434d11346144778e385ac18d417859 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250114 02:53:41.613724 1577 tablet_service.cc:1467] Processing CreateTablet for tablet edafd45551ec43f9a5b750c1e54c62f6 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250114 02:53:41.613724 1726 tablet_service.cc:1467] Processing CreateTablet for tablet d43434bae9eb4b2fb3699343149bb385 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250114 02:53:41.620335 1776 ts_tablet_manager.cc:1397] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.182s user 0.014s sys 0.010s
I20250114 02:53:41.621112 1575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8434d11346144778e385ac18d417859. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.623641 1776 raft_consensus.cc:357] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:41.624925 1776 raft_consensus.cc:383] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:41.625238 1779 tablet_bootstrap.cc:492] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:41.627416 1778 tablet_bootstrap.cc:492] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:41.629256 1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edafd45551ec43f9a5b750c1e54c62f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.636047 1645 tablet_service.cc:1467] Processing CreateTablet for tablet edafd45551ec43f9a5b750c1e54c62f6 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250114 02:53:41.636376 1644 tablet_service.cc:1467] Processing CreateTablet for tablet f8434d11346144778e385ac18d417859 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250114 02:53:41.637318 1645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edafd45551ec43f9a5b750c1e54c62f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.640472 1576 tablet_service.cc:1467] Processing CreateTablet for tablet d43434bae9eb4b2fb3699343149bb385 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250114 02:53:41.641938 1576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43434bae9eb4b2fb3699343149bb385. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.625257 1776 raft_consensus.cc:738] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:41.646385 1568 tablet_service.cc:1467] Processing CreateTablet for tablet 4a25a35a51714764a2d368283f31d19a (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250114 02:53:41.644208 1644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8434d11346144778e385ac18d417859. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.634120 1778 tablet_bootstrap.cc:654] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:41.648838 1647 tablet_service.cc:1467] Processing CreateTablet for tablet d43434bae9eb4b2fb3699343149bb385 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250114 02:53:41.648690 1776 consensus_queue.cc:260] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:41.650888 1779 tablet_bootstrap.cc:654] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:41.655753 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 4a25a35a51714764a2d368283f31d19a (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250114 02:53:41.656950 1650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a25a35a51714764a2d368283f31d19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.657457 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 0ecc21b2d4ec4b0aab2d25073324044b (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250114 02:53:41.658636 1646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ecc21b2d4ec4b0aab2d25073324044b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.663038 1722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8434d11346144778e385ac18d417859. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.664716 1647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43434bae9eb4b2fb3699343149bb385. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.643182 1721 tablet_service.cc:1467] Processing CreateTablet for tablet fb007689a5d64ee2a8fa67adad56de19 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250114 02:53:41.614372 1719 tablet_service.cc:1467] Processing CreateTablet for tablet 4a25a35a51714764a2d368283f31d19a (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250114 02:53:41.678107 1574 tablet_service.cc:1467] Processing CreateTablet for tablet 9fec1da7d2f8404bb83e600bc99eb944 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250114 02:53:41.679798 1574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fec1da7d2f8404bb83e600bc99eb944. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.680104 1575 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce7223c99df448483de68f3463ba115 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250114 02:53:41.681345 1575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ce7223c99df448483de68f3463ba115. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.687122 1568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a25a35a51714764a2d368283f31d19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.687721 1724 tablet_service.cc:1467] Processing CreateTablet for tablet 0ecc21b2d4ec4b0aab2d25073324044b (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250114 02:53:41.689222 1720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef83c2f4c7494118a6564eaca6b7799d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.690618 1725 tablet_service.cc:1467] Processing CreateTablet for tablet 9fec1da7d2f8404bb83e600bc99eb944 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250114 02:53:41.692202 1725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fec1da7d2f8404bb83e600bc99eb944. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.695271 1726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43434bae9eb4b2fb3699343149bb385. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.695785 1573 tablet_service.cc:1467] Processing CreateTablet for tablet 01ce953d409541bb85818e3dcf36541b (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250114 02:53:41.696892 1718 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce7223c99df448483de68f3463ba115 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250114 02:53:41.697943 1718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ce7223c99df448483de68f3463ba115. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.701632 1721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb007689a5d64ee2a8fa67adad56de19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.702875 1719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a25a35a51714764a2d368283f31d19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.696905 1573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ce953d409541bb85818e3dcf36541b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.709867 1687 heartbeater.cc:502] Master 127.0.106.126:37383 was elected leader, sending a full tablet report...
I20250114 02:53:41.713954 1781 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:41.714810 1724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ecc21b2d4ec4b0aab2d25073324044b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.714488 1781 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:41.678046 1572 tablet_service.cc:1467] Processing CreateTablet for tablet 0ecc21b2d4ec4b0aab2d25073324044b (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250114 02:53:41.727233 1572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ecc21b2d4ec4b0aab2d25073324044b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.733045 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 9fec1da7d2f8404bb83e600bc99eb944 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250114 02:53:41.735713 1643 tablet_service.cc:1467] Processing CreateTablet for tablet e36c50a200b24d17bb9a68ddc9dfa709 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250114 02:53:41.737609 1574 tablet_service.cc:1467] Processing CreateTablet for tablet daf552d03bce48dda4e5b9772287649a (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250114 02:53:41.743407 1571 tablet_service.cc:1467] Processing CreateTablet for tablet f4346db2c8a3424ba9562e58d10084db (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250114 02:53:41.744699 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce7223c99df448483de68f3463ba115 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250114 02:53:41.746980 1781 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:41.748373 1776 ts_tablet_manager.cc:1428] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.128s user 0.010s sys 0.007s
I20250114 02:53:41.753597 1577 tablet_service.cc:1467] Processing CreateTablet for tablet e36c50a200b24d17bb9a68ddc9dfa709 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250114 02:53:41.763814 1717 tablet_service.cc:1467] Processing CreateTablet for tablet f4346db2c8a3424ba9562e58d10084db (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250114 02:53:41.767815 1726 tablet_service.cc:1467] Processing CreateTablet for tablet 01ce953d409541bb85818e3dcf36541b (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250114 02:53:41.768918 1778 tablet_bootstrap.cc:492] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:41.831048 1778 ts_tablet_manager.cc:1397] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.204s user 0.007s sys 0.009s
I20250114 02:53:41.831338 1726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ce953d409541bb85818e3dcf36541b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.832523 1719 tablet_service.cc:1467] Processing CreateTablet for tablet 0b78459d23a744819d55e5f50de8174f (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250114 02:53:41.776433 1574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf552d03bce48dda4e5b9772287649a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.833629 1719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b78459d23a744819d55e5f50de8174f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.778403 1649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ce7223c99df448483de68f3463ba115. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.834856 1571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4346db2c8a3424ba9562e58d10084db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.836156 1643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36c50a200b24d17bb9a68ddc9dfa709. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.781527 1650 tablet_service.cc:1467] Processing CreateTablet for tablet f4346db2c8a3424ba9562e58d10084db (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250114 02:53:41.838825 1651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fec1da7d2f8404bb83e600bc99eb944. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.839450 1650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4346db2c8a3424ba9562e58d10084db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.783788 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 01ce953d409541bb85818e3dcf36541b (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250114 02:53:41.843362 1648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ce953d409541bb85818e3dcf36541b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.788964 1570 tablet_service.cc:1467] Processing CreateTablet for tablet 0b78459d23a744819d55e5f50de8174f (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250114 02:53:41.797224 1779 tablet_bootstrap.cc:492] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:41.802881 1721 tablet_service.cc:1467] Processing CreateTablet for tablet daf552d03bce48dda4e5b9772287649a (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250114 02:53:41.847205 1717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4346db2c8a3424ba9562e58d10084db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.847332 1779 ts_tablet_manager.cc:1397] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.229s user 0.027s sys 0.024s
I20250114 02:53:41.847726 1721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf552d03bce48dda4e5b9772287649a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.803139 1646 tablet_service.cc:1467] Processing CreateTablet for tablet c1f2bc8299cf4834bebd9aa5ae95a56e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250114 02:53:41.851281 1646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f2bc8299cf4834bebd9aa5ae95a56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.854864 1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36c50a200b24d17bb9a68ddc9dfa709. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.856426 1570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b78459d23a744819d55e5f50de8174f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.857599 1719 tablet_service.cc:1467] Processing CreateTablet for tablet ff0a423b0ba64289acb34e5c4d31aad0 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250114 02:53:41.858764 1719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff0a423b0ba64289acb34e5c4d31aad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.833659 1778 raft_consensus.cc:357] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:41.859898 1778 raft_consensus.cc:383] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:41.860253 1778 raft_consensus.cc:738] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:41.861068 1778 consensus_queue.cc:260] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:41.803577 1575 tablet_service.cc:1467] Processing CreateTablet for tablet e98ff91cae554f0f9347da0c886767ca (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250114 02:53:41.770996 1572 tablet_service.cc:1467] Processing CreateTablet for tablet c1f2bc8299cf4834bebd9aa5ae95a56e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250114 02:53:41.804486 1723 tablet_service.cc:1467] Processing CreateTablet for tablet e36c50a200b24d17bb9a68ddc9dfa709 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250114 02:53:41.806154 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 0b78459d23a744819d55e5f50de8174f (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250114 02:53:41.808619 1645 tablet_service.cc:1467] Processing CreateTablet for tablet e98ff91cae554f0f9347da0c886767ca (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
W20250114 02:53:41.810591 1688 tablet.cc:2367] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 02:53:41.810659 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:53:41.811589 1569 tablet_service.cc:1467] Processing CreateTablet for tablet ff0a423b0ba64289acb34e5c4d31aad0 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250114 02:53:41.871491 1647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b78459d23a744819d55e5f50de8174f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.850122 1779 raft_consensus.cc:357] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:41.878734 1779 raft_consensus.cc:383] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:41.879066 1779 raft_consensus.cc:738] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:41.818061 1722 tablet_service.cc:1467] Processing CreateTablet for tablet c1f2bc8299cf4834bebd9aa5ae95a56e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250114 02:53:41.820578 1652 tablet_service.cc:1467] Processing CreateTablet for tablet ff0a423b0ba64289acb34e5c4d31aad0 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250114 02:53:41.828569 1776 tablet_bootstrap.cc:492] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:41.865180 1576 tablet_service.cc:1467] Processing CreateTablet for tablet 4d6f63bdbf49400aae5fb92a633411db (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250114 02:53:41.879770 1779 consensus_queue.cc:260] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:41.869108 1723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36c50a200b24d17bb9a68ddc9dfa709. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.869841 1572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f2bc8299cf4834bebd9aa5ae95a56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.881350 1575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e98ff91cae554f0f9347da0c886767ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.883754 1576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d6f63bdbf49400aae5fb92a633411db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.886011 1776 tablet_bootstrap.cc:654] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:41.813990 1644 tablet_service.cc:1467] Processing CreateTablet for tablet daf552d03bce48dda4e5b9772287649a (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250114 02:53:41.908066 1651 tablet_service.cc:1467] Processing CreateTablet for tablet c62fd5040e324f7091a420da4d3ae67d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250114 02:53:41.909309 1651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62fd5040e324f7091a420da4d3ae67d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.911195 1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff0a423b0ba64289acb34e5c4d31aad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.918644 1569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff0a423b0ba64289acb34e5c4d31aad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.936826 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 4d6f63bdbf49400aae5fb92a633411db (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250114 02:53:41.939782 1645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e98ff91cae554f0f9347da0c886767ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.948096 1722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f2bc8299cf4834bebd9aa5ae95a56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.948706 1724 tablet_service.cc:1467] Processing CreateTablet for tablet cab6c0814b8241c99bdeb347d765f929 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250114 02:53:41.948773 1721 tablet_service.cc:1467] Processing CreateTablet for tablet 903edcc3b2294723af07e77c238bd3c8 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250114 02:53:41.949944 1724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab6c0814b8241c99bdeb347d765f929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.955271 1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d6f63bdbf49400aae5fb92a633411db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.942144 1651 tablet_service.cc:1467] Processing CreateTablet for tablet cab6c0814b8241c99bdeb347d765f929 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250114 02:53:41.960299 1651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab6c0814b8241c99bdeb347d765f929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.972530 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:41.973361 1736 raft_consensus.cc:2463] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:41.972420 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:41.977393 1628 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:41.978168 1781 raft_consensus.cc:2798] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:41.978736 1781 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:41.979126 1781 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:41.964322 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 903edcc3b2294723af07e77c238bd3c8 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250114 02:53:41.988571 1645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 903edcc3b2294723af07e77c238bd3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.989570 1722 tablet_service.cc:1467] Processing CreateTablet for tablet c62fd5040e324f7091a420da4d3ae67d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250114 02:53:41.987490 1781 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:41.991923 1612 heartbeater.cc:502] Master 127.0.106.126:37383 was elected leader, sending a full tablet report...
I20250114 02:53:41.992694 1781 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:41.993832 1722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62fd5040e324f7091a420da4d3ae67d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:41.866624 1725 tablet_service.cc:1467] Processing CreateTablet for tablet 4d6f63bdbf49400aae5fb92a633411db (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250114 02:53:42.014775 1720 tablet_service.cc:1467] Processing CreateTablet for tablet e98ff91cae554f0f9347da0c886767ca (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250114 02:53:42.018178 1718 tablet_service.cc:1467] Processing CreateTablet for tablet 464b2086919241de965a1ac17809451c (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
W20250114 02:53:41.978255 1628 leader_election.cc:343] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:42.031886 1721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 903edcc3b2294723af07e77c238bd3c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:53:42.032371 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 54145us
I20250114 02:53:42.035388 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:42.033154 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 464b2086919241de965a1ac17809451c (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250114 02:53:42.038743 1644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf552d03bce48dda4e5b9772287649a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.039462 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:42.040012 1736 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:42.042208 1576 tablet_service.cc:1467] Processing CreateTablet for tablet 903edcc3b2294723af07e77c238bd3c8 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250114 02:53:42.043449 1576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 903edcc3b2294723af07e77c238bd3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.044703 1572 tablet_service.cc:1467] Processing CreateTablet for tablet c62fd5040e324f7091a420da4d3ae67d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250114 02:53:42.045779 1572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62fd5040e324f7091a420da4d3ae67d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.046144 1736 raft_consensus.cc:2463] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:42.047819 1628 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:42.049043 1761 heartbeater.cc:502] Master 127.0.106.126:37383 was elected leader, sending a full tablet report...
I20250114 02:53:42.049104 1781 raft_consensus.cc:2798] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:42.049305 1577 tablet_service.cc:1467] Processing CreateTablet for tablet 464b2086919241de965a1ac17809451c (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250114 02:53:42.049690 1781 raft_consensus.cc:695] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:42.050629 1781 consensus_queue.cc:237] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.051635 1779 ts_tablet_manager.cc:1428] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.204s user 0.052s sys 0.045s
I20250114 02:53:42.004526 1568 tablet_service.cc:1467] Processing CreateTablet for tablet cab6c0814b8241c99bdeb347d765f929 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250114 02:53:42.056622 1725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d6f63bdbf49400aae5fb92a633411db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.059506 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdaf3f35acd45469c23cec4d4112b11 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250114 02:53:42.060704 1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fdaf3f35acd45469c23cec4d4112b11. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:53:42.061817 1628 leader_election.cc:343] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:42.064392 1573 tablet_service.cc:1467] Processing CreateTablet for tablet 6b73d9b5d01245f8be76a02efe7e5eca (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250114 02:53:42.066533 1574 tablet_service.cc:1467] Processing CreateTablet for tablet 9222bb8af9ab4f35a3fa735c302f11ac (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250114 02:53:42.071391 1776 tablet_bootstrap.cc:492] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:42.071853 1776 ts_tablet_manager.cc:1397] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.243s user 0.019s sys 0.026s
I20250114 02:53:42.074397 1776 raft_consensus.cc:357] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.075590 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 6b73d9b5d01245f8be76a02efe7e5eca (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250114 02:53:42.075882 1776 raft_consensus.cc:383] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.076260 1776 raft_consensus.cc:738] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.076471 1720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e98ff91cae554f0f9347da0c886767ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.076928 1776 consensus_queue.cc:260] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.077584 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 9222bb8af9ab4f35a3fa735c302f11ac (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250114 02:53:42.050549 1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 464b2086919241de965a1ac17809451c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.079083 1568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab6c0814b8241c99bdeb347d765f929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.081256 1776 ts_tablet_manager.cc:1428] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250114 02:53:42.086012 1643 tablet_service.cc:1467] Processing CreateTablet for tablet 5e00ae680c2d4c5cbb5154723228286f (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250114 02:53:42.086757 1778 ts_tablet_manager.cc:1428] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.255s user 0.003s sys 0.014s
I20250114 02:53:42.087689 1779 tablet_bootstrap.cc:492] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:42.087823 1778 tablet_bootstrap.cc:492] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:42.093726 1573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b73d9b5d01245f8be76a02efe7e5eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.094646 1726 tablet_service.cc:1467] Processing CreateTablet for tablet 88a6663722394089be1417d1cf01653a (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250114 02:53:42.098443 1778 tablet_bootstrap.cc:654] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.098865 1776 tablet_bootstrap.cc:492] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:42.100298 1649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 464b2086919241de965a1ac17809451c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.103029 1718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 464b2086919241de965a1ac17809451c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.107059 1726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a6663722394089be1417d1cf01653a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.107514 1569 tablet_service.cc:1467] Processing CreateTablet for tablet 44b972e53ce14bd0825dc78f9c5c1a27 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250114 02:53:42.111891 1571 tablet_service.cc:1467] Processing CreateTablet for tablet 5e00ae680c2d4c5cbb5154723228286f (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250114 02:53:42.114900 1574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9222bb8af9ab4f35a3fa735c302f11ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.118865 1650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b73d9b5d01245f8be76a02efe7e5eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.118865 1569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44b972e53ce14bd0825dc78f9c5c1a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.120581 1643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e00ae680c2d4c5cbb5154723228286f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.123858 1571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e00ae680c2d4c5cbb5154723228286f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.129827 1721 tablet_service.cc:1467] Processing CreateTablet for tablet 6b73d9b5d01245f8be76a02efe7e5eca (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250114 02:53:42.137549 1779 tablet_bootstrap.cc:654] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.139006 1570 tablet_service.cc:1467] Processing CreateTablet for tablet 88a6663722394089be1417d1cf01653a (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250114 02:53:42.140208 1570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a6663722394089be1417d1cf01653a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.058444 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 88a6663722394089be1417d1cf01653a (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250114 02:53:42.142867 1643 tablet_service.cc:1467] Processing CreateTablet for tablet 44b972e53ce14bd0825dc78f9c5c1a27 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250114 02:53:42.146975 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7e023d568848b9861c2976a138d461 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250114 02:53:42.150147 1722 tablet_service.cc:1467] Processing CreateTablet for tablet 9222bb8af9ab4f35a3fa735c302f11ac (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250114 02:53:42.184648 1788 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:42.185230 1788 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.199998 1788 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:42.206502 1721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b73d9b5d01245f8be76a02efe7e5eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.208199 1778 tablet_bootstrap.cc:492] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:42.208699 1778 ts_tablet_manager.cc:1397] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.121s user 0.017s sys 0.025s
I20250114 02:53:42.213006 1778 raft_consensus.cc:357] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.213936 1778 raft_consensus.cc:383] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.214476 1778 raft_consensus.cc:738] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.213279 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 328b539725d54455aadcb0e76e17b4d6 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250114 02:53:42.226346 1781 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:42.226629 1717 tablet_service.cc:1467] Processing CreateTablet for tablet 5e00ae680c2d4c5cbb5154723228286f (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250114 02:53:42.226886 1781 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.227751 1778 consensus_queue.cc:260] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.229420 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:42.229880 1778 ts_tablet_manager.cc:1428] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
W20250114 02:53:42.230728 1628 leader_election.cc:343] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:42.231570 1781 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:42.232904 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:53:42.234033 1628 leader_election.cc:343] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:42.205627 1651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9222bb8af9ab4f35a3fa735c302f11ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.234414 1628 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:42.235343 1781 raft_consensus.cc:2743] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:42.236141 1643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44b972e53ce14bd0825dc78f9c5c1a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.238823 1647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a6663722394089be1417d1cf01653a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.242782 1778 tablet_bootstrap.cc:492] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:42.248821 1788 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:42.249336 1788 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.251199 1788 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:42.249001 1646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a7e023d568848b9861c2976a138d461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.214941 1722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9222bb8af9ab4f35a3fa735c302f11ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.255041 1717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e00ae680c2d4c5cbb5154723228286f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.257004 1776 tablet_bootstrap.cc:654] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.258860 1648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 328b539725d54455aadcb0e76e17b4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.261694 1723 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdaf3f35acd45469c23cec4d4112b11 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250114 02:53:42.262987 1723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fdaf3f35acd45469c23cec4d4112b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.264719 1571 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdaf3f35acd45469c23cec4d4112b11 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250114 02:53:42.266063 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd3f8af715b4ed8a97965a52b089a21 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250114 02:53:42.267251 1645 tablet_service.cc:1467] Processing CreateTablet for tablet f37e0491ebc34f29b68a6425e22a3a2d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250114 02:53:42.276727 1645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f37e0491ebc34f29b68a6425e22a3a2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.282498 1572 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7e023d568848b9861c2976a138d461 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250114 02:53:42.285526 1778 tablet_bootstrap.cc:654] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.286860 1719 tablet_service.cc:1467] Processing CreateTablet for tablet 44b972e53ce14bd0825dc78f9c5c1a27 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250114 02:53:42.287638 1571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fdaf3f35acd45469c23cec4d4112b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.292420 1720 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7e023d568848b9861c2976a138d461 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250114 02:53:42.297726 1720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a7e023d568848b9861c2976a138d461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.311942 1719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44b972e53ce14bd0825dc78f9c5c1a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.315255 1779 tablet_bootstrap.cc:492] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:42.315753 1779 ts_tablet_manager.cc:1397] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.228s user 0.015s sys 0.024s
I20250114 02:53:42.324689 1779 raft_consensus.cc:357] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.325418 1779 raft_consensus.cc:383] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.325707 1779 raft_consensus.cc:738] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.326418 1779 consensus_queue.cc:260] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.328773 1779 ts_tablet_manager.cc:1428] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250114 02:53:42.332731 1779 tablet_bootstrap.cc:492] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:42.331300 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 6bae187c925c49dd9bb36f34be387836 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250114 02:53:42.337215 1719 tablet_service.cc:1467] Processing CreateTablet for tablet f37e0491ebc34f29b68a6425e22a3a2d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250114 02:53:42.337756 1575 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd3f8af715b4ed8a97965a52b089a21 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250114 02:53:42.340034 1779 tablet_bootstrap.cc:654] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.338322 1719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f37e0491ebc34f29b68a6425e22a3a2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.356277 1575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cd3f8af715b4ed8a97965a52b089a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.342228 1577 tablet_service.cc:1467] Processing CreateTablet for tablet 2d291f7c9c5547cf9006ed2dcc6b048e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250114 02:53:42.365412 1574 tablet_service.cc:1467] Processing CreateTablet for tablet 30486d50dbd94c78a14bad5b98494276 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250114 02:53:42.366356 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 8c49b6c0ab1d49079166b6a4cdf59896 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250114 02:53:42.369844 1643 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4107caddb14178ae1ca233127cc4a3 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250114 02:53:42.374688 1725 tablet_service.cc:1467] Processing CreateTablet for tablet 328b539725d54455aadcb0e76e17b4d6 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250114 02:53:42.379094 1485 catalog_manager.cc:5526] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:42.374368 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 4876e2bdbafa4d8684adc1348f1aeb80 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250114 02:53:42.381949 1725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 328b539725d54455aadcb0e76e17b4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.379611 1726 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd3f8af715b4ed8a97965a52b089a21 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250114 02:53:42.394309 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 2fccc76f52ed4204b3a8b03af5f65b1d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250114 02:53:42.390188 1649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cd3f8af715b4ed8a97965a52b089a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.401795 1726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cd3f8af715b4ed8a97965a52b089a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.391642 1576 tablet_service.cc:1467] Processing CreateTablet for tablet 328b539725d54455aadcb0e76e17b4d6 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250114 02:53:42.402647 1643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4107caddb14178ae1ca233127cc4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.403841 1648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bae187c925c49dd9bb36f34be387836. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.405974 1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4876e2bdbafa4d8684adc1348f1aeb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.406260 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 2d291f7c9c5547cf9006ed2dcc6b048e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250114 02:53:42.413213 1574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30486d50dbd94c78a14bad5b98494276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.414909 1576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 328b539725d54455aadcb0e76e17b4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.386997 1573 tablet_service.cc:1467] Processing CreateTablet for tablet f37e0491ebc34f29b68a6425e22a3a2d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250114 02:53:42.416730 1724 tablet_service.cc:1467] Processing CreateTablet for tablet 5d77d39f9a4b4a57ae17db8e50237cdb (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250114 02:53:42.417860 1573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f37e0491ebc34f29b68a6425e22a3a2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.418134 1724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d77d39f9a4b4a57ae17db8e50237cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.419756 1572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a7e023d568848b9861c2976a138d461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.421685 1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d291f7c9c5547cf9006ed2dcc6b048e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.422660 1721 tablet_service.cc:1467] Processing CreateTablet for tablet 4876e2bdbafa4d8684adc1348f1aeb80 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250114 02:53:42.424157 1721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4876e2bdbafa4d8684adc1348f1aeb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.438750 1779 tablet_bootstrap.cc:492] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:42.439249 1779 ts_tablet_manager.cc:1397] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.107s user 0.007s sys 0.021s
I20250114 02:53:42.441929 1779 raft_consensus.cc:357] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.446862 1646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d291f7c9c5547cf9006ed2dcc6b048e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.451612 1779 raft_consensus.cc:383] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.451929 1779 raft_consensus.cc:738] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.453501 1779 consensus_queue.cc:260] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.454463 1644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c49b6c0ab1d49079166b6a4cdf59896. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.456918 1779 ts_tablet_manager.cc:1428] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20250114 02:53:42.457752 1779 tablet_bootstrap.cc:492] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:42.466279 1779 tablet_bootstrap.cc:654] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.410161 1718 tablet_service.cc:1467] Processing CreateTablet for tablet 2d291f7c9c5547cf9006ed2dcc6b048e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250114 02:53:42.466781 1647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fccc76f52ed4204b3a8b03af5f65b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.472378 1778 tablet_bootstrap.cc:492] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:42.472129 1569 tablet_service.cc:1467] Processing CreateTablet for tablet 2fccc76f52ed4204b3a8b03af5f65b1d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250114 02:53:42.472926 1778 ts_tablet_manager.cc:1397] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.242s user 0.016s sys 0.017s
I20250114 02:53:42.473598 1569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fccc76f52ed4204b3a8b03af5f65b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.482460 1776 tablet_bootstrap.cc:492] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:42.394609 1568 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4107caddb14178ae1ca233127cc4a3 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250114 02:53:42.482903 1776 ts_tablet_manager.cc:1397] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.384s user 0.017s sys 0.004s
I20250114 02:53:42.489120 1568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4107caddb14178ae1ca233127cc4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.396375 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 30486d50dbd94c78a14bad5b98494276 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250114 02:53:42.493782 1717 tablet_service.cc:1467] Processing CreateTablet for tablet 30486d50dbd94c78a14bad5b98494276 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
W20250114 02:53:42.498531 1628 consensus_peers.cc:487] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 -> Peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Couldn't send request to peer 62daa58e8ff64698bb46ea3fd5290cdd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:42.519140 1778 raft_consensus.cc:357] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.519867 1778 raft_consensus.cc:383] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.520195 1778 raft_consensus.cc:738] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.505421 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 024e8460e81348268cc6fd408ece2a86 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250114 02:53:42.507001 1721 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4107caddb14178ae1ca233127cc4a3 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250114 02:53:42.510856 1573 tablet_service.cc:1467] Processing CreateTablet for tablet 6bae187c925c49dd9bb36f34be387836 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250114 02:53:42.512506 1576 tablet_service.cc:1467] Processing CreateTablet for tablet 07160adeb64840179d1c1529d86c94ed (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250114 02:53:42.486496 1718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d291f7c9c5547cf9006ed2dcc6b048e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.568316 1717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30486d50dbd94c78a14bad5b98494276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.522549 1778 consensus_queue.cc:260] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.527073 1650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30486d50dbd94c78a14bad5b98494276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.529788 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 07160adeb64840179d1c1529d86c94ed (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250114 02:53:42.531706 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 11bad528717d449380b9e81d7cf948d3 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250114 02:53:42.542471 1786 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:42.558737 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:42.560204 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:53:42.581954 1553 leader_election.cc:343] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:53:42.582743 1553 leader_election.cc:343] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:42.595465 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:42.595587 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:42.596356 1735 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
W20250114 02:53:42.596750 1553 leader_election.cc:343] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:42.597355 1553 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:42.597671 1721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4107caddb14178ae1ca233127cc4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.598026 1788 raft_consensus.cc:2743] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:42.599054 1553 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:42.599694 1723 tablet_service.cc:1467] Processing CreateTablet for tablet 2fccc76f52ed4204b3a8b03af5f65b1d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250114 02:53:42.601730 1723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fccc76f52ed4204b3a8b03af5f65b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.604869 1573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bae187c925c49dd9bb36f34be387836. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.605890 1571 tablet_service.cc:1467] Processing CreateTablet for tablet 4876e2bdbafa4d8684adc1348f1aeb80 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250114 02:53:42.599547 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 985a08c846764478aaeb97aa5c283bf9 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250114 02:53:42.599813 1788 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:42.606941 1788 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:42.607227 1788 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:42.607719 1722 tablet_service.cc:1467] Processing CreateTablet for tablet 6bae187c925c49dd9bb36f34be387836 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250114 02:53:42.608851 1722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bae187c925c49dd9bb36f34be387836. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.610463 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 7828a26fd7794b8a9ae9737e193416d2 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250114 02:53:42.611303 1574 tablet_service.cc:1467] Processing CreateTablet for tablet 024e8460e81348268cc6fd408ece2a86 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250114 02:53:42.613396 1577 tablet_service.cc:1467] Processing CreateTablet for tablet 11bad528717d449380b9e81d7cf948d3 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250114 02:53:42.615026 1788 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.608487 1776 raft_consensus.cc:357] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.620550 1776 raft_consensus.cc:383] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.620836 1776 raft_consensus.cc:738] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.622407 1788 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:42.573483 1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 024e8460e81348268cc6fd408ece2a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.626091 1645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bad528717d449380b9e81d7cf948d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.575546 1779 tablet_bootstrap.cc:492] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:42.626662 1779 ts_tablet_manager.cc:1397] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.169s user 0.015s sys 0.002s
I20250114 02:53:42.626480 1717 tablet_service.cc:1467] Processing CreateTablet for tablet 8c49b6c0ab1d49079166b6a4cdf59896 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250114 02:53:42.627626 1717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c49b6c0ab1d49079166b6a4cdf59896. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.629393 1779 raft_consensus.cc:357] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.630143 1779 raft_consensus.cc:383] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.576655 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 5d77d39f9a4b4a57ae17db8e50237cdb (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250114 02:53:42.579766 1786 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.630810 1651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07160adeb64840179d1c1529d86c94ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.632462 1786 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:42.632710 1574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 024e8460e81348268cc6fd408ece2a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.632822 1720 tablet_service.cc:1467] Processing CreateTablet for tablet 7828a26fd7794b8a9ae9737e193416d2 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250114 02:53:42.633268 1647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d77d39f9a4b4a57ae17db8e50237cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.633877 1720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7828a26fd7794b8a9ae9737e193416d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.634917 1571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4876e2bdbafa4d8684adc1348f1aeb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.635200 1644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 985a08c846764478aaeb97aa5c283bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.635840 1568 tablet_service.cc:1467] Processing CreateTablet for tablet c818f667d1314fc68ee42a2fd1c5a627 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250114 02:53:42.636938 1568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c818f667d1314fc68ee42a2fd1c5a627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.639065 1576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07160adeb64840179d1c1529d86c94ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.642714 1779 raft_consensus.cc:738] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.646911 1779 consensus_queue.cc:260] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.624111 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:42.650828 1776 consensus_queue.cc:260] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.653249 1776 ts_tablet_manager.cc:1428] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.136s user 0.004s sys 0.002s
I20250114 02:53:42.654150 1776 tablet_bootstrap.cc:492] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:42.654644 1725 tablet_service.cc:1467] Processing CreateTablet for tablet 024e8460e81348268cc6fd408ece2a86 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250114 02:53:42.656132 1725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 024e8460e81348268cc6fd408ece2a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.658020 1779 ts_tablet_manager.cc:1428] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20250114 02:53:42.661620 1572 tablet_service.cc:1467] Processing CreateTablet for tablet 8c49b6c0ab1d49079166b6a4cdf59896 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250114 02:53:42.661624 1570 tablet_service.cc:1467] Processing CreateTablet for tablet 5d77d39f9a4b4a57ae17db8e50237cdb (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250114 02:53:42.617439 1718 tablet_service.cc:1467] Processing CreateTablet for tablet 07160adeb64840179d1c1529d86c94ed (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250114 02:53:42.666278 1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bad528717d449380b9e81d7cf948d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.666947 1718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07160adeb64840179d1c1529d86c94ed. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:53:42.666962 1553 leader_election.cc:343] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:42.625137 1726 tablet_service.cc:1467] Processing CreateTablet for tablet c818f667d1314fc68ee42a2fd1c5a627 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250114 02:53:42.670174 1726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c818f667d1314fc68ee42a2fd1c5a627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.565361 1643 tablet_service.cc:1467] Processing CreateTablet for tablet c818f667d1314fc68ee42a2fd1c5a627 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250114 02:53:42.643533 1648 tablet_service.cc:1467] Processing CreateTablet for tablet f704f7a8203f4a5fabb81d95f3a5f38e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250114 02:53:42.644390 1646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7828a26fd7794b8a9ae9737e193416d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.674815 1648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f704f7a8203f4a5fabb81d95f3a5f38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.677572 1781 consensus_queue.cc:1035] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:42.623122 1719 tablet_service.cc:1467] Processing CreateTablet for tablet 11bad528717d449380b9e81d7cf948d3 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250114 02:53:42.576311 1778 ts_tablet_manager.cc:1428] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.103s user 0.005s sys 0.000s
I20250114 02:53:42.623728 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:42.679157 1735 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:42.685189 1643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c818f667d1314fc68ee42a2fd1c5a627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.690265 1735 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:42.691707 1776 tablet_bootstrap.cc:654] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.629814 1724 tablet_service.cc:1467] Processing CreateTablet for tablet 985a08c846764478aaeb97aa5c283bf9 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250114 02:53:42.696475 1575 tablet_service.cc:1467] Processing CreateTablet for tablet 985a08c846764478aaeb97aa5c283bf9 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250114 02:53:42.697590 1575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 985a08c846764478aaeb97aa5c283bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.710515 1572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c49b6c0ab1d49079166b6a4cdf59896. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.702908 1569 tablet_service.cc:1467] Processing CreateTablet for tablet 7828a26fd7794b8a9ae9737e193416d2 (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250114 02:53:42.712402 1569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7828a26fd7794b8a9ae9737e193416d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.713639 1573 tablet_service.cc:1467] Processing CreateTablet for tablet f704f7a8203f4a5fabb81d95f3a5f38e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250114 02:53:42.714581 1573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f704f7a8203f4a5fabb81d95f3a5f38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.715145 1717 tablet_service.cc:1467] Processing CreateTablet for tablet f704f7a8203f4a5fabb81d95f3a5f38e (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250114 02:53:42.707120 1779 tablet_bootstrap.cc:492] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:42.730392 1570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d77d39f9a4b4a57ae17db8e50237cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.725833 1719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bad528717d449380b9e81d7cf948d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.745129 1778 tablet_bootstrap.cc:492] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:42.830878 1724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 985a08c846764478aaeb97aa5c283bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.832989 1778 tablet_bootstrap.cc:654] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.834959 1717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f704f7a8203f4a5fabb81d95f3a5f38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:42.840833 1779 tablet_bootstrap.cc:654] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.842770 1806 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:42.846133 1553 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:42.847128 1788 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:42.847005 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:42.852097 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:42.853139 1702 leader_election.cc:343] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:53:42.853868 1702 leader_election.cc:343] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:42.854319 1702 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:42.855173 1786 raft_consensus.cc:2743] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:42.848395 1806 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.864313 1806 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:42.865650 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:53:42.867386 1628 leader_election.cc:343] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:42.867394 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:42.868495 1587 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:42.869661 1628 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:42.870550 1806 raft_consensus.cc:2798] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:42.870913 1806 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:42.871234 1806 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:53:42.874389 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:53:42.879801 1776 tablet_bootstrap.cc:492] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:42.880291 1776 ts_tablet_manager.cc:1397] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.226s user 0.057s sys 0.049s
I20250114 02:53:42.882787 1776 raft_consensus.cc:357] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.883515 1776 raft_consensus.cc:383] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.883795 1776 raft_consensus.cc:738] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.892511 1776 consensus_queue.cc:260] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.891166 1788 raft_consensus.cc:695] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:42.903263 1806 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.904121 1788 consensus_queue.cc:237] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.905046 1806 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:42.910526 1776 ts_tablet_manager.cc:1428] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.030s user 0.002s sys 0.006s
I20250114 02:53:42.911834 1776 tablet_bootstrap.cc:492] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:42.913038 1778 tablet_bootstrap.cc:492] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:42.913159 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:42.913491 1778 ts_tablet_manager.cc:1397] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.171s user 0.019s sys 0.034s
I20250114 02:53:42.913666 1587 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:42.914070 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
W20250114 02:53:42.915995 1628 leader_election.cc:343] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:42.915966 1778 raft_consensus.cc:357] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.916751 1778 raft_consensus.cc:383] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.917069 1778 raft_consensus.cc:738] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.917748 1778 consensus_queue.cc:260] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.920019 1778 ts_tablet_manager.cc:1428] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250114 02:53:42.920904 1778 tablet_bootstrap.cc:492] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:42.924043 1484 catalog_manager.cc:5526] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:42.925058 1779 tablet_bootstrap.cc:492] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:42.925246 1587 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:42.925515 1779 ts_tablet_manager.cc:1397] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.231s user 0.018s sys 0.017s
I20250114 02:53:42.927238 1776 tablet_bootstrap.cc:654] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.927955 1779 raft_consensus.cc:357] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.928555 1779 raft_consensus.cc:383] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.928788 1779 raft_consensus.cc:738] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.928882 1778 tablet_bootstrap.cc:654] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.929572 1779 consensus_queue.cc:260] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.932215 1628 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:42.933519 1806 raft_consensus.cc:2798] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:42.934063 1779 ts_tablet_manager.cc:1428] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20250114 02:53:42.934024 1806 raft_consensus.cc:695] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:42.934937 1779 tablet_bootstrap.cc:492] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:42.935006 1806 consensus_queue.cc:237] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.949633 1779 tablet_bootstrap.cc:654] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.950546 1776 tablet_bootstrap.cc:492] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:42.951146 1776 ts_tablet_manager.cc:1397] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.004s
I20250114 02:53:42.953019 1778 tablet_bootstrap.cc:492] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:42.953539 1778 ts_tablet_manager.cc:1397] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.012s
I20250114 02:53:42.954665 1776 raft_consensus.cc:357] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.955327 1776 raft_consensus.cc:383] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.955610 1776 raft_consensus.cc:738] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.956274 1776 consensus_queue.cc:260] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.956363 1778 raft_consensus.cc:357] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.957358 1778 raft_consensus.cc:383] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.957697 1778 raft_consensus.cc:738] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.957897 1779 tablet_bootstrap.cc:492] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:42.958354 1779 ts_tablet_manager.cc:1397] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.002s
I20250114 02:53:42.958752 1776 ts_tablet_manager.cc:1428] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250114 02:53:42.958662 1778 consensus_queue.cc:260] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.959754 1776 tablet_bootstrap.cc:492] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:42.961037 1779 raft_consensus.cc:357] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.960700 1484 catalog_manager.cc:5526] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:42.961750 1779 raft_consensus.cc:383] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.962051 1779 raft_consensus.cc:738] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.962960 1779 consensus_queue.cc:260] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.966105 1776 tablet_bootstrap.cc:654] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.965143 1778 ts_tablet_manager.cc:1428] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.011s user 0.000s sys 0.005s
I20250114 02:53:42.968885 1806 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:42.969384 1806 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.970161 1779 ts_tablet_manager.cc:1428] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20250114 02:53:42.970988 1806 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:42.971167 1779 tablet_bootstrap.cc:492] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:42.972582 1776 tablet_bootstrap.cc:492] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:42.972476 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:42.973042 1776 ts_tablet_manager.cc:1397] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20250114 02:53:42.972914 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:42.973474 1735 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:42.973651 1778 tablet_bootstrap.cc:492] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
W20250114 02:53:42.973677 1628 leader_election.cc:343] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:42.977456 1779 tablet_bootstrap.cc:654] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.979545 1778 tablet_bootstrap.cc:654] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.981570 1776 raft_consensus.cc:357] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.982414 1776 raft_consensus.cc:383] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:42.982864 1776 raft_consensus.cc:738] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:42.983867 1776 consensus_queue.cc:260] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:42.977286 1628 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:42.987169 1776 ts_tablet_manager.cc:1428] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.013s user 0.004s sys 0.003s
I20250114 02:53:42.987731 1806 raft_consensus.cc:2798] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:42.988234 1806 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:42.988488 1776 tablet_bootstrap.cc:492] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:42.988896 1806 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:42.996090 1776 tablet_bootstrap.cc:654] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:42.999367 1779 tablet_bootstrap.cc:492] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:42.999792 1779 ts_tablet_manager.cc:1397] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.006s
I20250114 02:53:43.000344 1806 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.003111 1806 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.003611 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:43.003993 1778 tablet_bootstrap.cc:492] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:43.004436 1778 ts_tablet_manager.cc:1397] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.006s
W20250114 02:53:43.005148 1628 leader_election.cc:343] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.005609 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:43.006055 1776 tablet_bootstrap.cc:492] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:43.006146 1735 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.006548 1776 ts_tablet_manager.cc:1397] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20250114 02:53:43.006752 1778 raft_consensus.cc:357] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.007445 1778 raft_consensus.cc:383] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.007736 1778 raft_consensus.cc:738] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.002254 1779 raft_consensus.cc:357] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.009178 1779 raft_consensus.cc:383] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.009634 1779 raft_consensus.cc:738] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.009179 1776 raft_consensus.cc:357] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.010145 1776 raft_consensus.cc:383] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.010462 1776 raft_consensus.cc:738] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.010450 1779 consensus_queue.cc:260] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.011407 1776 consensus_queue.cc:260] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.012609 1779 ts_tablet_manager.cc:1428] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20250114 02:53:43.008376 1778 consensus_queue.cc:260] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.013563 1776 ts_tablet_manager.cc:1428] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250114 02:53:43.014667 1778 ts_tablet_manager.cc:1428] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20250114 02:53:43.014684 1776 tablet_bootstrap.cc:492] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:43.015467 1806 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.015857 1806 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.017645 1806 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.018666 1778 tablet_bootstrap.cc:492] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:43.019235 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:43.020543 1628 leader_election.cc:343] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.021308 1776 tablet_bootstrap.cc:654] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.024765 1778 tablet_bootstrap.cc:654] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.027639 1735 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:43.028867 1779 tablet_bootstrap.cc:492] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:43.029399 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.030222 1736 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:43.028654 1628 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:43.031610 1806 raft_consensus.cc:2798] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:43.031982 1806 raft_consensus.cc:695] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:43.032171 1628 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:43.032704 1806 consensus_queue.cc:237] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.033411 1815 raft_consensus.cc:2798] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:43.033735 1815 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:43.034018 1815 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.034967 1779 tablet_bootstrap.cc:654] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.043103 1815 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.046511 1814 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.048673 1814 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.049777 1778 tablet_bootstrap.cc:492] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:43.050237 1778 ts_tablet_manager.cc:1397] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.003s
I20250114 02:53:43.051028 1776 tablet_bootstrap.cc:492] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:43.051213 1814 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.052927 1778 raft_consensus.cc:357] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.053592 1778 raft_consensus.cc:383] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.053846 1778 raft_consensus.cc:738] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.051443 1776 ts_tablet_manager.cc:1397] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.005s
I20250114 02:53:43.054216 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:43.054585 1778 consensus_queue.cc:260] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.053851 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.056768 1778 ts_tablet_manager.cc:1428] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:53:43.056617 1776 raft_consensus.cc:357] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.057214 1776 raft_consensus.cc:383] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.057448 1776 raft_consensus.cc:738] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.057597 1778 tablet_bootstrap.cc:492] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
W20250114 02:53:43.058665 1628 leader_election.cc:343] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.058763 1776 consensus_queue.cc:260] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:43.059403 1628 leader_election.cc:343] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.060433 1484 catalog_manager.cc:5526] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:43.061628 1628 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:43.062354 1806 raft_consensus.cc:2743] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:43.064335 1779 tablet_bootstrap.cc:492] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:43.064749 1779 ts_tablet_manager.cc:1397] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.017s
I20250114 02:53:43.065623 1815 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.061295 1776 ts_tablet_manager.cc:1428] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20250114 02:53:43.066141 1778 tablet_bootstrap.cc:654] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.066715 1776 tablet_bootstrap.cc:492] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:43.066980 1779 raft_consensus.cc:357] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.067656 1779 raft_consensus.cc:383] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.067909 1779 raft_consensus.cc:738] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.068531 1779 consensus_queue.cc:260] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.070516 1779 ts_tablet_manager.cc:1428] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20250114 02:53:43.071424 1788 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.071528 1779 tablet_bootstrap.cc:492] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:43.071846 1788 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.072345 1776 tablet_bootstrap.cc:654] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.073573 1788 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.075706 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.077940 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:43.078017 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:43.078590 1736 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.078862 1778 tablet_bootstrap.cc:492] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:43.078791 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.079339 1778 ts_tablet_manager.cc:1397] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.003s
W20250114 02:53:43.081646 1553 leader_election.cc:343] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.081727 1778 raft_consensus.cc:357] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.082448 1778 raft_consensus.cc:383] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.082728 1778 raft_consensus.cc:738] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.083287 1778 consensus_queue.cc:260] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:43.082464 1553 leader_election.cc:343] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.085011 1553 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:43.086282 1788 raft_consensus.cc:2743] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:53:43.089061 1628 leader_election.cc:343] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.090822 1778 ts_tablet_manager.cc:1428] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250114 02:53:43.091696 1778 tablet_bootstrap.cc:492] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:43.092221 1736 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:43.093218 1628 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:43.093900 1815 raft_consensus.cc:2798] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:43.094318 1815 raft_consensus.cc:695] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:43.095250 1815 consensus_queue.cc:237] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.098708 1779 tablet_bootstrap.cc:654] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.100870 1587 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.115175 1776 tablet_bootstrap.cc:492] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:43.115309 1778 tablet_bootstrap.cc:654] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.115644 1776 ts_tablet_manager.cc:1397] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.049s user 0.017s sys 0.008s
I20250114 02:53:43.118075 1776 raft_consensus.cc:357] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.118845 1776 raft_consensus.cc:383] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.119158 1776 raft_consensus.cc:738] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.119972 1776 consensus_queue.cc:260] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.122469 1786 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.123325 1786 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.124398 1776 ts_tablet_manager.cc:1428] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250114 02:53:43.125772 1776 tablet_bootstrap.cc:492] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:43.127245 1786 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:43.134919 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.135830 1661 raft_consensus.cc:2463] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:43.137529 1702 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:43.138624 1786 raft_consensus.cc:2798] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:43.139110 1786 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:43.139532 1776 tablet_bootstrap.cc:654] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.139531 1786 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.142843 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:43.146656 1702 leader_election.cc:343] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.148882 1811 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:43.149533 1811 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.144371 1483 catalog_manager.cc:5526] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:43.152674 1779 tablet_bootstrap.cc:492] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:43.153049 1779 ts_tablet_manager.cc:1397] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.082s user 0.012s sys 0.018s
I20250114 02:53:43.153712 1811 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.154376 1778 tablet_bootstrap.cc:492] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:43.154811 1778 ts_tablet_manager.cc:1397] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.002s
I20250114 02:53:43.155478 1779 raft_consensus.cc:357] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.155951 1779 raft_consensus.cc:383] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.156107 1779 raft_consensus.cc:738] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.157228 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.156950 1779 consensus_queue.cc:260] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.158866 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.157853 1778 raft_consensus.cc:357] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.160048 1778 raft_consensus.cc:383] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.160423 1778 raft_consensus.cc:738] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.161118 1778 consensus_queue.cc:260] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.163237 1553 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:43.167284 1788 raft_consensus.cc:2743] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:43.176479 1836 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.179690 1778 ts_tablet_manager.cc:1428] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20250114 02:53:43.180742 1778 tablet_bootstrap.cc:492] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:43.178004 1836 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.184875 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:43.185544 1587 raft_consensus.cc:2388] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 1: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:43.185374 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.187271 1836 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:43.187383 1702 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:43.173897 1786 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.190304 1836 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.191308 1838 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.193351 1779 ts_tablet_manager.cc:1428] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.040s user 0.017s sys 0.016s
I20250114 02:53:43.194119 1779 tablet_bootstrap.cc:492] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:43.190721 1836 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.196916 1778 tablet_bootstrap.cc:654] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.199748 1779 tablet_bootstrap.cc:654] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.201200 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
W20250114 02:53:43.202394 1702 leader_election.cc:343] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.205050 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:43.205792 1587 raft_consensus.cc:2463] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:43.206653 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:43.207326 1661 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.208508 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.209538 1662 raft_consensus.cc:2463] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:43.211000 1702 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:43.212119 1786 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:43.212749 1786 raft_consensus.cc:2798] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:43.213117 1786 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:43.213474 1786 raft_consensus.cc:3054] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.216306 1661 raft_consensus.cc:2463] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:43.217478 1702 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:43.207314 1836 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:43.205080 1776 tablet_bootstrap.cc:492] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:43.219252 1776 ts_tablet_manager.cc:1397] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.094s user 0.016s sys 0.008s
I20250114 02:53:43.219403 1836 raft_consensus.cc:2798] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:43.220002 1836 raft_consensus.cc:695] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:43.221519 1836 consensus_queue.cc:237] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.226928 1776 raft_consensus.cc:357] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.227720 1776 raft_consensus.cc:383] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.228003 1776 raft_consensus.cc:738] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.231946 1779 tablet_bootstrap.cc:492] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:43.232254 1779 ts_tablet_manager.cc:1397] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.038s user 0.021s sys 0.013s
I20250114 02:53:43.233798 1779 raft_consensus.cc:357] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.234225 1779 raft_consensus.cc:383] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.234426 1779 raft_consensus.cc:738] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.234925 1779 consensus_queue.cc:260] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.234040 1483 catalog_manager.cc:5526] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:43.235549 1836 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.235929 1836 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.236567 1779 ts_tablet_manager.cc:1428] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250114 02:53:43.237313 1779 tablet_bootstrap.cc:492] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:43.238137 1836 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:43.238893 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:43.239706 1587 raft_consensus.cc:2463] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:43.241195 1702 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:43.241556 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.241963 1779 tablet_bootstrap.cc:654] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.242199 1661 raft_consensus.cc:2463] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:43.242539 1839 raft_consensus.cc:2798] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:43.243072 1839 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:43.243707 1839 raft_consensus.cc:3054] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.245726 1838 raft_consensus.cc:2743] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:43.246667 1776 consensus_queue.cc:260] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.252599 1776 ts_tablet_manager.cc:1428] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20250114 02:53:43.253623 1776 tablet_bootstrap.cc:492] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:43.256712 1779 tablet_bootstrap.cc:492] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:43.257091 1779 ts_tablet_manager.cc:1397] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.011s
I20250114 02:53:43.259810 1779 raft_consensus.cc:357] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.260609 1779 raft_consensus.cc:383] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.260849 1779 raft_consensus.cc:738] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.261461 1779 consensus_queue.cc:260] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.263455 1779 ts_tablet_manager.cc:1428] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250114 02:53:43.264207 1779 tablet_bootstrap.cc:492] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:43.270805 1779 tablet_bootstrap.cc:654] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.270388 1815 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.271554 1815 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.273775 1815 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.274798 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.274739 1786 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.275429 1736 raft_consensus.cc:2463] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:43.276575 1628 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:43.277217 1815 raft_consensus.cc:2798] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:43.277520 1815 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:43.277855 1815 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.281607 1836 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.281744 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:43.281883 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:43.282260 1586 raft_consensus.cc:3054] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.282016 1836 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:43.287204 1628 leader_election.cc:343] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.288524 1836 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:43.292531 1815 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.276961 1786 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:43.295464 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:43.296010 1661 raft_consensus.cc:3054] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.296454 1839 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.298432 1839 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:43.299809 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:43.300097 1815 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.300209 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:43.301082 1660 raft_consensus.cc:3054] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.301272 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:43.301853 1736 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.301776 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.302986 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:43.303579 1585 raft_consensus.cc:3054] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.300501 1587 raft_consensus.cc:2463] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:43.305477 1661 raft_consensus.cc:2463] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
W20250114 02:53:43.305162 1702 leader_election.cc:343] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.307394 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:43.307993 1776 tablet_bootstrap.cc:654] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
W20250114 02:53:43.308517 1628 leader_election.cc:343] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.310221 1779 tablet_bootstrap.cc:492] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:43.309875 1660 raft_consensus.cc:2463] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:43.310668 1779 ts_tablet_manager.cc:1397] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.028s
I20250114 02:53:43.309875 1585 raft_consensus.cc:2463] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:43.310472 1702 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:43.311762 1736 raft_consensus.cc:2463] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:43.311889 1702 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:43.312533 1786 raft_consensus.cc:2798] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:43.312681 1628 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:43.312963 1786 raft_consensus.cc:695] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:43.313287 1586 raft_consensus.cc:2463] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:43.313606 1778 tablet_bootstrap.cc:492] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:43.314008 1778 ts_tablet_manager.cc:1397] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.133s user 0.036s sys 0.043s
I20250114 02:53:43.313699 1786 consensus_queue.cc:237] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.314182 1815 raft_consensus.cc:2798] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:43.314839 1815 raft_consensus.cc:695] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:43.315716 1815 consensus_queue.cc:237] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.317106 1702 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:43.316394 1839 raft_consensus.cc:2798] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:43.318328 1839 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:43.318873 1839 raft_consensus.cc:3054] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.318665 1778 raft_consensus.cc:357] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.317680 1779 raft_consensus.cc:357] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.319418 1778 raft_consensus.cc:383] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.319598 1779 raft_consensus.cc:383] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.319705 1778 raft_consensus.cc:738] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.319883 1779 raft_consensus.cc:738] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.321233 1836 raft_consensus.cc:2798] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:43.321717 1836 raft_consensus.cc:695] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:43.321848 1779 consensus_queue.cc:260] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.322464 1836 consensus_queue.cc:237] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.320343 1778 consensus_queue.cc:260] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.329468 1778 ts_tablet_manager.cc:1428] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.015s user 0.003s sys 0.003s
I20250114 02:53:43.331537 1839 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.335328 1811 consensus_queue.cc:1035] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:43.336520 1839 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:43.332373 1778 tablet_bootstrap.cc:492] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:43.343776 1779 ts_tablet_manager.cc:1428] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.033s user 0.005s sys 0.007s
I20250114 02:53:43.346143 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:43.347280 1779 tablet_bootstrap.cc:492] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:43.346259 1485 catalog_manager.cc:5526] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:43.347980 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:43.349014 1586 raft_consensus.cc:3054] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:53:43.356858 1702 leader_election.cc:343] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.361068 1778 tablet_bootstrap.cc:654] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.361092 1815 consensus_queue.cc:1035] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:43.366762 1483 catalog_manager.cc:5526] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:43.373673 1586 raft_consensus.cc:2463] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:43.375752 1702 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:43.377039 1836 raft_consensus.cc:2798] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:43.377571 1836 raft_consensus.cc:695] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:43.378597 1836 consensus_queue.cc:237] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.374455 1779 tablet_bootstrap.cc:654] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.383432 1847 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.384011 1847 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.386737 1847 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:43.388531 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.388628 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:43.392949 1702 leader_election.cc:343] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:53:43.393569 1702 leader_election.cc:343] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.393940 1702 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:43.395345 1660 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.406078 1778 tablet_bootstrap.cc:492] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:43.406607 1778 ts_tablet_manager.cc:1397] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.074s user 0.018s sys 0.016s
I20250114 02:53:43.397341 1847 raft_consensus.cc:2743] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:43.429296 1776 tablet_bootstrap.cc:492] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:43.446662 1776 ts_tablet_manager.cc:1397] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.193s user 0.021s sys 0.024s
I20250114 02:53:43.455961 1779 tablet_bootstrap.cc:492] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:43.456439 1779 ts_tablet_manager.cc:1397] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.112s user 0.020s sys 0.005s
I20250114 02:53:43.458025 1778 raft_consensus.cc:357] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.458513 1778 raft_consensus.cc:383] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.459054 1778 raft_consensus.cc:738] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.459617 1778 consensus_queue.cc:260] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.463809 1811 consensus_queue.cc:1035] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:43.465289 1778 ts_tablet_manager.cc:1428] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.058s user 0.006s sys 0.000s
I20250114 02:53:43.466089 1811 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.466770 1814 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.468336 1779 raft_consensus.cc:357] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.468966 1788 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.471774 1779 raft_consensus.cc:383] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.471432 1776 raft_consensus.cc:357] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.472134 1779 raft_consensus.cc:738] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.472247 1776 raft_consensus.cc:383] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.469200 1811 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.472244 1788 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.472878 1779 consensus_queue.cc:260] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.473995 1811 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.474277 1788 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.475134 1779 ts_tablet_manager.cc:1428] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.018s user 0.003s sys 0.002s
I20250114 02:53:43.476027 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.476733 1736 raft_consensus.cc:2388] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:43.476512 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.476874 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.477344 1735 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:43.477481 1662 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:43.471729 1814 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.476544 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.470352 1483 catalog_manager.cc:5526] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:43.478395 1553 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:43.479571 1814 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.479878 1553 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:43.469803 1778 tablet_bootstrap.cc:492] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:43.479722 1483 catalog_manager.cc:5526] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:43.481556 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:43.480912 1788 raft_consensus.cc:2798] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:43.483583 1811 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.483646 1788 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:43.484125 1788 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.489306 1806 consensus_queue.cc:1035] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:43.491338 1788 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.493505 1788 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.493934 1788 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.494297 1778 tablet_bootstrap.cc:654] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.477649 1779 tablet_bootstrap.cc:492] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:43.501466 1776 raft_consensus.cc:738] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.507659 1776 consensus_queue.cc:260] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.506088 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.513635 1776 ts_tablet_manager.cc:1428] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.067s user 0.006s sys 0.000s
I20250114 02:53:43.514780 1776 tablet_bootstrap.cc:492] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:43.510022 1628 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:43.516793 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:43.517556 1661 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.522783 1779 tablet_bootstrap.cc:654] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.494336 1788 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.524963 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:43.504612 1811 raft_consensus.cc:2743] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:43.527691 1735 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.528659 1814 raft_consensus.cc:2743] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:43.528100 1815 consensus_queue.cc:1035] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:43.539948 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.544576 1661 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:43.546480 1553 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:43.547246 1858 raft_consensus.cc:2798] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:43.552771 1778 tablet_bootstrap.cc:492] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:43.553201 1778 ts_tablet_manager.cc:1397] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.014s
I20250114 02:53:43.556113 1815 consensus_queue.cc:1035] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:43.556778 1778 raft_consensus.cc:357] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.557574 1788 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.557580 1778 raft_consensus.cc:383] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.559218 1778 raft_consensus.cc:738] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.559780 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.562278 1553 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:43.563361 1811 raft_consensus.cc:2743] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:43.560767 1778 consensus_queue.cc:260] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.570323 1735 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:43.571849 1778 ts_tablet_manager.cc:1428] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.018s user 0.006s sys 0.002s
I20250114 02:53:43.572467 1858 raft_consensus.cc:695] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:43.573662 1858 consensus_queue.cc:237] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.578647 1778 tablet_bootstrap.cc:492] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:43.580842 1776 tablet_bootstrap.cc:654] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.593953 1585 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.596964 1485 catalog_manager.cc:5526] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:43.606246 1814 consensus_queue.cc:1035] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:43.615796 1814 consensus_queue.cc:1035] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:43.650424 1778 tablet_bootstrap.cc:654] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.650864 1776 tablet_bootstrap.cc:492] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:43.651463 1779 tablet_bootstrap.cc:492] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:43.651932 1779 ts_tablet_manager.cc:1397] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.174s user 0.015s sys 0.040s
I20250114 02:53:43.651290 1776 ts_tablet_manager.cc:1397] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.137s user 0.022s sys 0.002s
I20250114 02:53:43.656282 1778 tablet_bootstrap.cc:492] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:43.657215 1778 ts_tablet_manager.cc:1397] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.079s user 0.010s sys 0.005s
I20250114 02:53:43.657541 1776 raft_consensus.cc:357] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.658190 1776 raft_consensus.cc:383] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.658440 1776 raft_consensus.cc:738] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.659212 1776 consensus_queue.cc:260] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.660043 1778 raft_consensus.cc:357] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.660723 1778 raft_consensus.cc:383] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.660951 1776 ts_tablet_manager.cc:1428] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250114 02:53:43.661294 1779 raft_consensus.cc:357] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.661979 1779 raft_consensus.cc:383] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.662235 1779 raft_consensus.cc:738] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.662858 1779 consensus_queue.cc:260] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.665001 1779 ts_tablet_manager.cc:1428] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250114 02:53:43.665745 1585 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.665930 1779 tablet_bootstrap.cc:492] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:43.660964 1778 raft_consensus.cc:738] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.669999 1778 consensus_queue.cc:260] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.672037 1778 ts_tablet_manager.cc:1428] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250114 02:53:43.672448 1779 tablet_bootstrap.cc:654] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.672880 1778 tablet_bootstrap.cc:492] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:43.677317 1776 tablet_bootstrap.cc:492] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:43.680023 1847 consensus_queue.cc:1035] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:43.689239 1778 tablet_bootstrap.cc:654] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.693061 1776 tablet_bootstrap.cc:654] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.704802 1779 tablet_bootstrap.cc:492] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:43.705277 1779 ts_tablet_manager.cc:1397] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.011s
I20250114 02:53:43.707726 1779 raft_consensus.cc:357] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.708460 1779 raft_consensus.cc:383] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.708519 1814 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.708895 1779 raft_consensus.cc:738] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.708940 1814 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.709717 1779 consensus_queue.cc:260] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.710928 1814 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.712307 1779 ts_tablet_manager.cc:1428] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250114 02:53:43.713152 1779 tablet_bootstrap.cc:492] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:43.712705 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.713618 1736 raft_consensus.cc:2463] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:43.714769 1628 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:43.714790 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:43.716678 1814 raft_consensus.cc:2798] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250114 02:53:43.716792 1628 leader_election.cc:343] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.717034 1814 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:43.717373 1814 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.719040 1786 consensus_queue.cc:1035] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:43.726599 1779 tablet_bootstrap.cc:654] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.727874 1776 tablet_bootstrap.cc:492] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:43.728294 1776 ts_tablet_manager.cc:1397] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.013s
I20250114 02:53:43.728176 1836 consensus_queue.cc:1035] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:43.731964 1776 raft_consensus.cc:357] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.732787 1776 raft_consensus.cc:383] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.733176 1776 raft_consensus.cc:738] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.733214 1858 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.733887 1858 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.734155 1776 consensus_queue.cc:260] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.735859 1778 tablet_bootstrap.cc:492] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:43.735910 1858 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.736287 1778 ts_tablet_manager.cc:1397] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.064s user 0.020s sys 0.000s
I20250114 02:53:43.737457 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.737661 1776 ts_tablet_manager.cc:1428] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20250114 02:53:43.739559 1553 leader_election.cc:343] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.739565 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.742031 1778 raft_consensus.cc:357] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:43.742291 1553 leader_election.cc:343] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.742741 1778 raft_consensus.cc:383] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.742887 1553 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:43.743069 1778 raft_consensus.cc:738] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.744069 1778 consensus_queue.cc:260] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.745810 1863 raft_consensus.cc:2743] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:43.746158 1776 tablet_bootstrap.cc:492] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:43.751729 1778 ts_tablet_manager.cc:1428] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20250114 02:53:43.751714 1806 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.752461 1806 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.753204 1778 tablet_bootstrap.cc:492] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:43.755558 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:43.756033 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.756626 1736 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
W20250114 02:53:43.756783 1628 leader_election.cc:343] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.754336 1806 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.758181 1628 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:43.760365 1806 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:43.760687 1806 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:43.760990 1806 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.766249 1814 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.769901 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:43.773160 1806 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.774796 1806 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.780026 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:43.780578 1736 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.781126 1778 tablet_bootstrap.cc:654] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.781505 1776 tablet_bootstrap.cc:654] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.781342 1838 consensus_queue.cc:1035] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:53:43.782874 1628 leader_election.cc:343] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.783538 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:43.787621 1806 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:53:43.787783 1628 leader_election.cc:343] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.788296 1806 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.790025 1806 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.791543 1736 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:43.794178 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:43.794940 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.795550 1814 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.796397 1628 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:43.797511 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:43.798300 1735 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.796630 1779 tablet_bootstrap.cc:492] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:43.800621 1779 ts_tablet_manager.cc:1397] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.088s user 0.017s sys 0.012s
I20250114 02:53:43.801359 1585 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.798372 1814 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:43.803166 1814 raft_consensus.cc:695] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:43.804231 1814 consensus_queue.cc:237] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.807503 1838 consensus_queue.cc:1035] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:43.812179 1779 raft_consensus.cc:357] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.812526 1806 consensus_queue.cc:1035] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:43.812759 1779 raft_consensus.cc:383] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.814276 1779 raft_consensus.cc:738] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.810528 1587 raft_consensus.cc:2388] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 1: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:43.816174 1779 consensus_queue.cc:260] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.816707 1628 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:43.817615 1806 raft_consensus.cc:3054] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.818722 1779 ts_tablet_manager.cc:1428] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20250114 02:53:43.819608 1779 tablet_bootstrap.cc:492] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:43.815940 1735 raft_consensus.cc:2463] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:43.824374 1628 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:43.824823 1806 raft_consensus.cc:2743] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:43.825276 1815 raft_consensus.cc:2798] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:43.825723 1815 raft_consensus.cc:695] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:43.827302 1776 tablet_bootstrap.cc:492] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:43.826834 1815 consensus_queue.cc:237] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.827653 1776 ts_tablet_manager.cc:1397] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.082s user 0.014s sys 0.013s
I20250114 02:53:43.834847 1838 consensus_queue.cc:1035] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:43.841061 1776 raft_consensus.cc:357] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.841871 1776 raft_consensus.cc:383] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.842171 1776 raft_consensus.cc:738] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.853732 1776 consensus_queue.cc:260] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.866243 1858 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.869979 1776 ts_tablet_manager.cc:1428] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20250114 02:53:43.871096 1776 tablet_bootstrap.cc:492] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:43.868854 1858 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.876716 1858 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:43.878917 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.881022 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:43.881829 1735 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:43.885128 1838 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:53:43.885692 1553 leader_election.cc:343] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.888742 1778 tablet_bootstrap.cc:492] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:43.889171 1778 ts_tablet_manager.cc:1397] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.136s user 0.005s sys 0.010s
I20250114 02:53:43.885553 1838 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.891744 1778 raft_consensus.cc:357] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.893730 1778 raft_consensus.cc:383] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:43.894001 1778 raft_consensus.cc:738] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:43.886550 1553 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:43.895140 1858 raft_consensus.cc:2798] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:43.894730 1778 consensus_queue.cc:260] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.897662 1838 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:43.898161 1838 consensus_queue.cc:1035] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:43.898500 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:43.898662 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:43.897686 1778 ts_tablet_manager.cc:1428] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250114 02:53:43.901368 1778 tablet_bootstrap.cc:492] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
W20250114 02:53:43.900216 1702 leader_election.cc:343] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.895509 1858 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 02:53:43.906750 1702 leader_election.cc:343] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.907241 1702 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:43.908690 1836 raft_consensus.cc:2743] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:43.909322 1779 tablet_bootstrap.cc:654] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.910216 1858 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.920135 1838 consensus_queue.cc:1035] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 02:53:43.927829 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:53:43.928716 1776 tablet_bootstrap.cc:654] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:43.942879 1778 tablet_bootstrap.cc:654] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
W20250114 02:53:43.932364 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 66130.2us
I20250114 02:53:43.966311 1786 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.966931 1786 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:43.969197 1786 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:43.971621 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:43.972397 1586 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:43.975003 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:53:43.976765 1702 leader_election.cc:343] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:43.977386 1702 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:43.978381 1786 raft_consensus.cc:2798] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:43.978744 1786 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:43.979120 1786 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:43.996611 1889 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:43.997077 1889 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.000283 1889 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.003634 1779 tablet_bootstrap.cc:492] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:44.004122 1779 ts_tablet_manager.cc:1397] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.185s user 0.009s sys 0.009s
I20250114 02:53:43.997843 1858 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.004910 1786 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.014618 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:44.014526 1779 raft_consensus.cc:357] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.015149 1586 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.015322 1779 raft_consensus.cc:383] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.015673 1779 raft_consensus.cc:738] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.016296 1779 consensus_queue.cc:260] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.021307 1779 ts_tablet_manager.cc:1428] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20250114 02:53:44.022176 1779 tablet_bootstrap.cc:492] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:44.029587 1586 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:44.030113 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:44.024304 1786 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:53:44.035943 1702 leader_election.cc:343] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.043897 1778 tablet_bootstrap.cc:492] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:44.046936 1702 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:44.050812 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:44.051044 1779 tablet_bootstrap.cc:654] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.051473 1735 raft_consensus.cc:2463] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:44.056314 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:44.051111 1483 catalog_manager.cc:5526] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:44.052718 1858 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.052881 1838 raft_consensus.cc:2798] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:44.059561 1838 raft_consensus.cc:695] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
W20250114 02:53:44.059767 1553 leader_election.cc:343] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.060379 1838 consensus_queue.cc:237] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.054337 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:44.062770 1736 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.055699 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:44.056334 1778 ts_tablet_manager.cc:1397] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.155s user 0.014s sys 0.006s
I20250114 02:53:44.069361 1628 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:44.070158 1815 raft_consensus.cc:2798] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:44.070588 1815 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 02:53:44.070612 1628 leader_election.cc:343] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.071096 1815 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.071847 1736 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:44.072700 1863 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.074102 1863 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.080641 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:44.081259 1863 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.067121 1894 consensus_queue.cc:1035] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:53:44.081964 1553 leader_election.cc:343] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.086325 1778 raft_consensus.cc:357] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.087181 1778 raft_consensus.cc:383] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.087487 1778 raft_consensus.cc:738] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.090937 1776 tablet_bootstrap.cc:492] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:44.091383 1776 ts_tablet_manager.cc:1397] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.220s user 0.026s sys 0.005s
I20250114 02:53:44.092304 1553 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:44.092782 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:53:44.093825 1553 leader_election.cc:343] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.094215 1553 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:44.094532 1779 tablet_bootstrap.cc:492] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:44.094936 1779 ts_tablet_manager.cc:1397] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.073s user 0.013s sys 0.010s
I20250114 02:53:44.094341 1776 raft_consensus.cc:357] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.096282 1776 raft_consensus.cc:383] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.096490 1776 raft_consensus.cc:738] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.088173 1778 consensus_queue.cc:260] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.098719 1858 raft_consensus.cc:2798] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:44.099493 1858 raft_consensus.cc:695] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:44.098819 1815 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.099994 1778 ts_tablet_manager.cc:1428] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.036s user 0.005s sys 0.001s
I20250114 02:53:44.100363 1858 consensus_queue.cc:237] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.100750 1776 consensus_queue.cc:260] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.101261 1778 tablet_bootstrap.cc:492] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:44.102530 1776 ts_tablet_manager.cc:1428] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20250114 02:53:44.103324 1776 tablet_bootstrap.cc:492] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:44.103569 1483 catalog_manager.cc:5526] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:44.096896 1863 raft_consensus.cc:2743] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:44.121354 1779 raft_consensus.cc:357] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.122001 1779 raft_consensus.cc:383] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.122373 1779 raft_consensus.cc:738] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.123049 1779 consensus_queue.cc:260] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.125197 1779 ts_tablet_manager.cc:1428] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20250114 02:53:44.127715 1776 tablet_bootstrap.cc:654] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.129901 1779 tablet_bootstrap.cc:492] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:44.130391 1815 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.132923 1485 catalog_manager.cc:5526] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:44.159564 1483 catalog_manager.cc:5526] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:44.137528 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:44.262065 1858 consensus_queue.cc:1035] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:44.268584 1779 tablet_bootstrap.cc:654] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
W20250114 02:53:44.280830 1628 leader_election.cc:343] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:53:44.271571 1899 log.cc:926] Time spent T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
I20250114 02:53:44.200765 1838 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.285260 1778 tablet_bootstrap.cc:654] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.141742 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:44.220894 1898 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.288828 1898 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.290738 1898 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.294457 1779 tablet_bootstrap.cc:492] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:44.295019 1779 ts_tablet_manager.cc:1397] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.168s user 0.012s sys 0.006s
I20250114 02:53:44.298024 1779 raft_consensus.cc:357] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.298770 1779 raft_consensus.cc:383] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.299121 1779 raft_consensus.cc:738] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.286155 1838 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.299796 1779 consensus_queue.cc:260] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.287320 1890 consensus_queue.cc:1035] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20250114 02:53:44.306059 1838 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:44.309175 1779 ts_tablet_manager.cc:1428] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250114 02:53:44.287042 1735 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.310317 1779 tablet_bootstrap.cc:492] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:44.312947 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:44.313084 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:44.313426 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:44.305277 1898 consensus_queue.cc:1035] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
W20250114 02:53:44.315482 1628 consensus_peers.cc:487] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 -> Peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Couldn't send request to peer 62daa58e8ff64698bb46ea3fd5290cdd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250114 02:53:44.316823 1702 leader_election.cc:343] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.318024 1815 consensus_queue.cc:1035] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:44.318692 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:44.319309 1586 raft_consensus.cc:2463] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:44.323024 1702 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:44.323962 1836 raft_consensus.cc:2798] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:44.324311 1836 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:44.325821 1735 raft_consensus.cc:2463] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:44.329209 1836 raft_consensus.cc:3054] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.342461 1836 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.368528 1836 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:53:44.384258 1553 leader_election.cc:343] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.384752 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
W20250114 02:53:44.385027 1553 leader_election.cc:343] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.385293 1586 raft_consensus.cc:3054] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.385720 1553 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:44.385779 1628 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:44.387162 1858 raft_consensus.cc:2743] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:44.389811 1776 tablet_bootstrap.cc:492] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:44.390175 1776 ts_tablet_manager.cc:1397] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.287s user 0.047s sys 0.063s
I20250114 02:53:44.391901 1815 raft_consensus.cc:2798] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:44.393695 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:44.393893 1815 raft_consensus.cc:695] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:44.394510 1778 tablet_bootstrap.cc:492] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:44.394920 1778 ts_tablet_manager.cc:1397] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.294s user 0.014s sys 0.046s
I20250114 02:53:44.394882 1815 consensus_queue.cc:237] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.399289 1776 raft_consensus.cc:357] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.400233 1776 raft_consensus.cc:383] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.400509 1776 raft_consensus.cc:738] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.413865 1776 consensus_queue.cc:260] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.414461 1586 raft_consensus.cc:2463] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:44.415153 1778 raft_consensus.cc:357] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.415946 1778 raft_consensus.cc:383] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250114 02:53:44.416033 1702 leader_election.cc:343] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.416209 1778 raft_consensus.cc:738] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.417788 1778 consensus_queue.cc:260] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.418254 1702 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:44.419636 1779 tablet_bootstrap.cc:654] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.419667 1778 ts_tablet_manager.cc:1428] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.025s user 0.007s sys 0.004s
I20250114 02:53:44.420567 1786 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.421572 1484 catalog_manager.cc:5526] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:44.422281 1847 raft_consensus.cc:2798] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:44.422739 1847 raft_consensus.cc:695] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:44.423511 1847 consensus_queue.cc:237] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.420965 1786 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.426029 1786 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:53:44.421187 1628 consensus_peers.cc:487] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 -> Peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Couldn't send request to peer 62daa58e8ff64698bb46ea3fd5290cdd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:44.429270 1776 ts_tablet_manager.cc:1428] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.039s user 0.008s sys 0.000s
I20250114 02:53:44.420568 1778 tablet_bootstrap.cc:492] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:44.430032 1776 tablet_bootstrap.cc:492] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:44.436249 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:44.438257 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:44.455195 1779 tablet_bootstrap.cc:492] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:44.455572 1779 ts_tablet_manager.cc:1397] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.145s user 0.011s sys 0.007s
W20250114 02:53:44.455749 1702 leader_election.cc:343] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.458235 1779 raft_consensus.cc:357] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.458982 1779 raft_consensus.cc:383] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.459254 1779 raft_consensus.cc:738] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.459817 1779 consensus_queue.cc:260] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:44.460415 1702 leader_election.cc:343] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.460805 1702 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:44.461442 1786 raft_consensus.cc:2743] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:44.465750 1779 ts_tablet_manager.cc:1428] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250114 02:53:44.466701 1779 tablet_bootstrap.cc:492] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:44.466924 1776 tablet_bootstrap.cc:654] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.472163 1779 tablet_bootstrap.cc:654] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.471501 1485 catalog_manager.cc:5526] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:44.472623 1778 tablet_bootstrap.cc:654] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.479347 1776 tablet_bootstrap.cc:492] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:44.479864 1776 ts_tablet_manager.cc:1397] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.050s user 0.017s sys 0.000s
I20250114 02:53:44.479929 1778 tablet_bootstrap.cc:492] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:44.481295 1778 ts_tablet_manager.cc:1397] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.005s
I20250114 02:53:44.482138 1779 tablet_bootstrap.cc:492] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:44.482031 1776 raft_consensus.cc:357] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.482796 1776 raft_consensus.cc:383] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.482964 1779 ts_tablet_manager.cc:1397] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20250114 02:53:44.483124 1776 raft_consensus.cc:738] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.483453 1778 raft_consensus.cc:357] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.483824 1890 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.484117 1778 raft_consensus.cc:383] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.483886 1776 consensus_queue.cc:260] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.484388 1778 raft_consensus.cc:738] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.484314 1890 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.484961 1778 consensus_queue.cc:260] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.485607 1779 raft_consensus.cc:357] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.485991 1890 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.486169 1779 raft_consensus.cc:383] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.486419 1779 raft_consensus.cc:738] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.487378 1779 consensus_queue.cc:260] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.488221 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:44.489459 1779 ts_tablet_manager.cc:1428] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20250114 02:53:44.489892 1778 ts_tablet_manager.cc:1428] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250114 02:53:44.490674 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:44.491366 1732 raft_consensus.cc:2388] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:44.490689 1778 tablet_bootstrap.cc:492] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:44.492244 1628 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:44.493414 1894 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.496816 1778 tablet_bootstrap.cc:654] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.498870 1779 tablet_bootstrap.cc:492] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:44.499351 1894 raft_consensus.cc:2743] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:44.499684 1776 ts_tablet_manager.cc:1428] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250114 02:53:44.500510 1776 tablet_bootstrap.cc:492] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:44.502784 1786 consensus_queue.cc:1035] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:44.520288 1779 tablet_bootstrap.cc:654] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.534828 1776 tablet_bootstrap.cc:654] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.540462 1779 tablet_bootstrap.cc:492] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:44.540851 1779 ts_tablet_manager.cc:1397] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.017s
I20250114 02:53:44.542801 1778 tablet_bootstrap.cc:492] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:44.543354 1778 ts_tablet_manager.cc:1397] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.053s user 0.029s sys 0.020s
I20250114 02:53:44.543130 1779 raft_consensus.cc:357] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.543902 1779 raft_consensus.cc:383] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.544183 1779 raft_consensus.cc:738] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.544715 1779 consensus_queue.cc:260] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.545965 1778 raft_consensus.cc:357] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.546609 1779 ts_tablet_manager.cc:1428] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250114 02:53:44.546645 1778 raft_consensus.cc:383] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.547106 1778 raft_consensus.cc:738] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.547658 1778 consensus_queue.cc:260] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.548081 1779 tablet_bootstrap.cc:492] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:44.549934 1778 ts_tablet_manager.cc:1428] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20250114 02:53:44.552260 1702 consensus_peers.cc:487] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 -> Peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Couldn't send request to peer 1e5870f8269b49cf81dc39f1a5c8f5c1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:44.552724 1778 tablet_bootstrap.cc:492] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:44.562767 1779 tablet_bootstrap.cc:654] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.563439 1776 tablet_bootstrap.cc:492] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:44.563725 1776 ts_tablet_manager.cc:1397] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.063s user 0.014s sys 0.005s
I20250114 02:53:44.565153 1893 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.565672 1778 tablet_bootstrap.cc:654] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.565580 1893 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.566372 1776 raft_consensus.cc:357] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.567010 1776 raft_consensus.cc:383] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.567257 1776 raft_consensus.cc:738] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.567454 1893 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.567826 1776 consensus_queue.cc:260] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.568420 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:44.569119 1732 raft_consensus.cc:2463] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:44.570359 1776 ts_tablet_manager.cc:1428] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250114 02:53:44.571276 1778 tablet_bootstrap.cc:492] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:44.571739 1778 ts_tablet_manager.cc:1397] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20250114 02:53:44.572021 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:44.572445 1628 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:44.573252 1893 raft_consensus.cc:2798] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:44.573555 1893 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:44.573823 1893 raft_consensus.cc:3054] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.573848 1778 raft_consensus.cc:357] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.574529 1776 tablet_bootstrap.cc:492] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:44.577105 1779 tablet_bootstrap.cc:492] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:44.577533 1779 ts_tablet_manager.cc:1397] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.008s
I20250114 02:53:44.579710 1893 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.580296 1776 tablet_bootstrap.cc:654] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.579962 1779 raft_consensus.cc:357] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.580832 1779 raft_consensus.cc:383] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.581146 1779 raft_consensus.cc:738] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
W20250114 02:53:44.581336 1628 leader_election.cc:343] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.581899 1779 consensus_queue.cc:260] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.581511 1893 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.574543 1778 raft_consensus.cc:383] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.583449 1778 raft_consensus.cc:738] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.583968 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:44.584072 1778 consensus_queue.cc:260] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.584931 1586 raft_consensus.cc:3054] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.585055 1779 ts_tablet_manager.cc:1428] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250114 02:53:44.585515 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:44.586026 1732 raft_consensus.cc:3054] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.586406 1778 ts_tablet_manager.cc:1428] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20250114 02:53:44.587231 1778 tablet_bootstrap.cc:492] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:44.591754 1586 raft_consensus.cc:2463] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:44.594071 1779 tablet_bootstrap.cc:492] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:44.595921 1628 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:44.601720 1893 raft_consensus.cc:2798] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:44.603276 1893 raft_consensus.cc:695] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:44.603833 1779 tablet_bootstrap.cc:654] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.604220 1776 tablet_bootstrap.cc:492] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:44.607311 1776 ts_tablet_manager.cc:1397] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.012s
I20250114 02:53:44.604024 1893 consensus_queue.cc:237] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.608671 1732 raft_consensus.cc:2463] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:44.610952 1776 raft_consensus.cc:357] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.611418 1776 raft_consensus.cc:383] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.611653 1776 raft_consensus.cc:738] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.612164 1776 consensus_queue.cc:260] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.612452 1779 tablet_bootstrap.cc:492] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:44.612990 1779 ts_tablet_manager.cc:1397] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20250114 02:53:44.613695 1776 ts_tablet_manager.cc:1428] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250114 02:53:44.614327 1776 tablet_bootstrap.cc:492] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:44.606542 1778 tablet_bootstrap.cc:654] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.616529 1779 raft_consensus.cc:357] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.617341 1779 raft_consensus.cc:383] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.617663 1779 raft_consensus.cc:738] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.618479 1779 consensus_queue.cc:260] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.620175 1776 tablet_bootstrap.cc:654] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.625898 1779 ts_tablet_manager.cc:1428] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20250114 02:53:44.627012 1778 tablet_bootstrap.cc:492] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:44.627068 1779 tablet_bootstrap.cc:492] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:44.627593 1778 ts_tablet_manager.cc:1397] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.004s
I20250114 02:53:44.630014 1858 consensus_queue.cc:1035] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:44.630138 1778 raft_consensus.cc:357] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.630941 1778 raft_consensus.cc:383] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.631140 1863 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.631239 1778 raft_consensus.cc:738] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.631357 1838 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.632223 1778 consensus_queue.cc:260] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.632375 1838 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.634963 1838 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:44.639194 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:44.639791 1660 raft_consensus.cc:2463] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:44.631673 1863 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.640947 1893 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.641297 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:44.641929 1863 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.641618 1893 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.642943 1484 catalog_manager.cc:5526] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:44.644387 1893 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.644862 1702 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:44.645943 1839 raft_consensus.cc:2798] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:44.645815 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:44.646368 1778 ts_tablet_manager.cc:1428] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.018s user 0.004s sys 0.002s
I20250114 02:53:44.646978 1839 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:44.648095 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:44.648588 1839 raft_consensus.cc:3054] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.650468 1778 tablet_bootstrap.cc:492] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:44.652005 1893 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.652822 1893 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.654546 1893 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.651795 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:44.656318 1587 raft_consensus.cc:2388] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 1: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:44.651197 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:44.649487 1586 raft_consensus.cc:2463] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:44.659570 1553 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:44.660816 1898 raft_consensus.cc:2743] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:44.660743 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:44.661656 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:44.662333 1839 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.664353 1839 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:44.665079 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:44.660938 1628 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:44.665755 1587 raft_consensus.cc:3054] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.666787 1779 tablet_bootstrap.cc:654] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.666929 1894 raft_consensus.cc:3054] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.667196 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:44.670439 1660 raft_consensus.cc:3054] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.679921 1587 raft_consensus.cc:2463] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:44.679987 1628 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:44.680783 1776 tablet_bootstrap.cc:492] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:44.681175 1776 ts_tablet_manager.cc:1397] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.067s user 0.023s sys 0.000s
I20250114 02:53:44.682792 1702 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:44.683807 1839 raft_consensus.cc:2798] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:44.684300 1890 raft_consensus.cc:2743] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:44.684432 1839 raft_consensus.cc:695] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:44.685185 1839 consensus_queue.cc:237] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.690519 1776 raft_consensus.cc:357] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.691247 1776 raft_consensus.cc:383] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.691480 1779 tablet_bootstrap.cc:492] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:44.691556 1776 raft_consensus.cc:738] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.691929 1779 ts_tablet_manager.cc:1397] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.015s
I20250114 02:53:44.685206 1660 raft_consensus.cc:2463] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:44.696005 1894 raft_consensus.cc:2743] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:44.692487 1776 consensus_queue.cc:260] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.693640 1898 consensus_queue.cc:1035] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:44.696560 1778 tablet_bootstrap.cc:654] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.697023 1779 raft_consensus.cc:357] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.697666 1779 raft_consensus.cc:383] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.697924 1779 raft_consensus.cc:738] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.698491 1779 consensus_queue.cc:260] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.700536 1779 ts_tablet_manager.cc:1428] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250114 02:53:44.701627 1779 tablet_bootstrap.cc:492] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:44.700567 1776 ts_tablet_manager.cc:1428] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.019s user 0.003s sys 0.003s
I20250114 02:53:44.703579 1776 tablet_bootstrap.cc:492] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:44.707535 1893 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.707953 1893 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.712142 1778 tablet_bootstrap.cc:492] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:44.712555 1778 ts_tablet_manager.cc:1397] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.003s
I20250114 02:53:44.714064 1779 tablet_bootstrap.cc:654] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.714995 1778 raft_consensus.cc:357] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.715584 1778 raft_consensus.cc:383] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.715830 1778 raft_consensus.cc:738] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.716351 1778 consensus_queue.cc:260] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.716763 1893 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.718434 1778 ts_tablet_manager.cc:1428] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20250114 02:53:44.718351 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:44.719331 1778 tablet_bootstrap.cc:492] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:44.718993 1587 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:44.715386 1484 catalog_manager.cc:5526] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:44.723155 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:44.724576 1628 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:44.728121 1732 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:44.728775 1893 raft_consensus.cc:2798] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:44.729136 1893 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:44.729465 1893 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.732784 1587 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.736526 1778 tablet_bootstrap.cc:654] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.744359 1893 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.746081 1893 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.727016 1776 tablet_bootstrap.cc:654] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.747740 1898 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.748211 1898 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.749804 1779 tablet_bootstrap.cc:492] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:44.750217 1779 ts_tablet_manager.cc:1397] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.049s user 0.016s sys 0.020s
I20250114 02:53:44.752242 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:44.752575 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:44.752923 1732 raft_consensus.cc:2463] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:44.753295 1779 raft_consensus.cc:357] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.753921 1779 raft_consensus.cc:383] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.754016 1553 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:44.754215 1779 raft_consensus.cc:738] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.754922 1779 consensus_queue.cc:260] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.754909 1858 raft_consensus.cc:2798] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:44.755610 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:44.755875 1858 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:44.756227 1586 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.757340 1779 ts_tablet_manager.cc:1428] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250114 02:53:44.758098 1779 tablet_bootstrap.cc:492] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:44.756224 1858 raft_consensus.cc:3054] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:53:44.761927 1553 leader_election.cc:343] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.750180 1898 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.762717 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:44.763788 1779 tablet_bootstrap.cc:654] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.763847 1732 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.780320 1732 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:44.781709 1628 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:44.783367 1893 raft_consensus.cc:2798] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:44.783830 1893 raft_consensus.cc:695] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:44.784610 1893 consensus_queue.cc:237] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.786094 1586 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:44.792285 1898 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.792754 1898 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.794479 1898 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.788812 1858 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.795296 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:44.798935 1776 tablet_bootstrap.cc:492] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:44.798807 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:44.799401 1776 ts_tablet_manager.cc:1397] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.096s user 0.006s sys 0.012s
I20250114 02:53:44.800631 1553 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:44.801604 1898 raft_consensus.cc:2743] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:44.803622 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:44.801810 1858 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.804589 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:44.804522 1732 raft_consensus.cc:3054] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:53:44.805868 1553 leader_election.cc:343] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.808593 1839 raft_consensus.cc:491] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.809032 1839 raft_consensus.cc:513] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.810746 1839 leader_election.cc:290] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:44.812953 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:53:44.814280 1702 leader_election.cc:343] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.814630 1779 tablet_bootstrap.cc:492] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:44.815002 1858 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.815117 1779 ts_tablet_manager.cc:1397] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.057s user 0.020s sys 0.019s
I20250114 02:53:44.816211 1858 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.801824 1776 raft_consensus.cc:357] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.815573 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:44.817691 1776 raft_consensus.cc:383] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.817967 1587 raft_consensus.cc:2463] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:44.817986 1776 raft_consensus.cc:738] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.818284 1779 raft_consensus.cc:357] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.818900 1776 consensus_queue.cc:260] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.819232 1779 raft_consensus.cc:383] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.819175 1702 leader_election.cc:304] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:44.809450 1483 catalog_manager.cc:5526] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:44.820179 1839 raft_consensus.cc:2798] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:44.820519 1839 raft_consensus.cc:491] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:44.820823 1839 raft_consensus.cc:3054] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.821275 1776 ts_tablet_manager.cc:1428] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20250114 02:53:44.821127 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:44.822077 1776 tablet_bootstrap.cc:492] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:44.823465 1858 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:44.829525 1732 raft_consensus.cc:2463] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:44.831300 1553 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:44.832324 1858 raft_consensus.cc:2798] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:44.832830 1778 tablet_bootstrap.cc:492] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:44.833071 1858 raft_consensus.cc:695] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:44.833248 1778 ts_tablet_manager.cc:1397] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.114s user 0.008s sys 0.000s
I20250114 02:53:44.833802 1858 consensus_queue.cc:237] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.830919 1776 tablet_bootstrap.cc:654] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.837685 1778 raft_consensus.cc:357] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.838126 1778 raft_consensus.cc:383] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.838317 1778 raft_consensus.cc:738] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.838876 1778 consensus_queue.cc:260] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.840729 1776 tablet_bootstrap.cc:492] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:44.841056 1778 ts_tablet_manager.cc:1428] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250114 02:53:44.841104 1776 ts_tablet_manager.cc:1397] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.002s
I20250114 02:53:44.841477 1779 raft_consensus.cc:738] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.842022 1779 consensus_queue.cc:260] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.842550 1778 tablet_bootstrap.cc:492] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:44.845439 1839 raft_consensus.cc:513] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.847085 1839 leader_election.cc:290] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:44.844480 1776 raft_consensus.cc:357] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.847834 1776 raft_consensus.cc:383] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.848078 1776 raft_consensus.cc:738] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.847015 1483 catalog_manager.cc:5526] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:44.848644 1776 consensus_queue.cc:260] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.851016 1776 ts_tablet_manager.cc:1428] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20250114 02:53:44.843351 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:44.852021 1732 raft_consensus.cc:2388] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:44.852432 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:44.853142 1776 tablet_bootstrap.cc:492] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:44.855969 1778 tablet_bootstrap.cc:654] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.857280 1553 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:44.858562 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:44.859162 1587 raft_consensus.cc:3054] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.860426 1779 ts_tablet_manager.cc:1428] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.044s user 0.005s sys 0.000s
I20250114 02:53:44.862735 1779 tablet_bootstrap.cc:492] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:44.863097 1858 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:53:44.861032 1702 leader_election.cc:343] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.871158 1776 tablet_bootstrap.cc:654] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.877748 1858 raft_consensus.cc:2743] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:44.876176 1839 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.876003 1587 raft_consensus.cc:2463] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:44.879679 1702 leader_election.cc:304] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:44.880172 1778 tablet_bootstrap.cc:492] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:44.880571 1778 ts_tablet_manager.cc:1397] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.006s
I20250114 02:53:44.880836 1838 raft_consensus.cc:2798] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:44.881340 1838 raft_consensus.cc:695] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:44.882084 1838 consensus_queue.cc:237] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.882510 1893 consensus_queue.cc:1035] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:44.879102 1839 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.882529 1778 raft_consensus.cc:357] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.884137 1778 raft_consensus.cc:383] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.884311 1778 raft_consensus.cc:738] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.884773 1778 consensus_queue.cc:260] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.886562 1587 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.887743 1778 ts_tablet_manager.cc:1428] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20250114 02:53:44.888685 1778 tablet_bootstrap.cc:492] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:44.890304 1779 tablet_bootstrap.cc:654] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.896606 1776 tablet_bootstrap.cc:492] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:44.897228 1776 ts_tablet_manager.cc:1397] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.044s user 0.014s sys 0.000s
I20250114 02:53:44.900930 1776 raft_consensus.cc:357] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.901618 1776 raft_consensus.cc:383] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.901854 1776 raft_consensus.cc:738] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.902382 1776 consensus_queue.cc:260] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.903033 1839 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:44.904029 1778 tablet_bootstrap.cc:654] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.905454 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:44.905967 1585 raft_consensus.cc:2463] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:44.906805 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:44.910049 1776 ts_tablet_manager.cc:1428] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250114 02:53:44.910854 1776 tablet_bootstrap.cc:492] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
W20250114 02:53:44.912915 1702 leader_election.cc:343] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.913671 1702 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:44.914511 1839 raft_consensus.cc:2798] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:44.914749 1839 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:44.914940 1839 raft_consensus.cc:3054] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.917655 1776 tablet_bootstrap.cc:654] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:44.911983 1483 catalog_manager.cc:5526] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:44.919850 1894 consensus_queue.cc:1035] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:44.920732 1839 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.924109 1778 tablet_bootstrap.cc:492] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:44.924491 1778 ts_tablet_manager.cc:1397] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.000s
I20250114 02:53:44.937593 1894 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.944325 1894 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.944967 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:44.945197 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:44.945732 1586 raft_consensus.cc:3054] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.947176 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:44.948197 1702 leader_election.cc:343] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.948693 1894 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:44.950342 1628 leader_election.cc:343] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:44.951473 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:44.952023 1732 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:44.952877 1628 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:44.953008 1660 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.941725 1778 raft_consensus.cc:357] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.943104 1839 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:44.954064 1778 raft_consensus.cc:383] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.955143 1778 raft_consensus.cc:738] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.955426 1776 tablet_bootstrap.cc:492] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:44.955783 1776 ts_tablet_manager.cc:1397] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.019s
I20250114 02:53:44.956315 1778 consensus_queue.cc:260] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.957789 1779 tablet_bootstrap.cc:492] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:44.958070 1779 ts_tablet_manager.cc:1397] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.096s user 0.008s sys 0.009s
I20250114 02:53:44.954993 1894 raft_consensus.cc:2798] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:44.963961 1776 raft_consensus.cc:357] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.968009 1776 raft_consensus.cc:383] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.968298 1776 raft_consensus.cc:738] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.968883 1776 consensus_queue.cc:260] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.969229 1586 raft_consensus.cc:2463] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:44.967679 1779 raft_consensus.cc:357] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.969950 1779 raft_consensus.cc:383] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:44.970216 1779 raft_consensus.cc:738] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:44.970959 1776 ts_tablet_manager.cc:1428] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20250114 02:53:44.970742 1702 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:44.971527 1838 raft_consensus.cc:2798] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:44.971935 1838 raft_consensus.cc:695] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:44.966895 1778 ts_tablet_manager.cc:1428] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.042s user 0.004s sys 0.003s
I20250114 02:53:44.973217 1838 consensus_queue.cc:237] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.970747 1779 consensus_queue.cc:260] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:44.974134 1778 tablet_bootstrap.cc:492] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:44.975589 1776 tablet_bootstrap.cc:492] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:44.975579 1894 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:44.985803 1894 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:44.988958 1779 ts_tablet_manager.cc:1428] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.031s user 0.001s sys 0.005s
I20250114 02:53:44.989972 1779 tablet_bootstrap.cc:492] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:44.991180 1815 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:44.964314 1839 consensus_queue.cc:1035] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20250114 02:53:44.994009 1815 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.005499 1893 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.005787 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.005976 1893 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:45.007640 1628 leader_election.cc:343] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.007721 1893 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.005493 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.008503 1483 catalog_manager.cc:5526] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:45.001857 1815 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:45.011006 1628 leader_election.cc:343] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.011526 1628 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:45.009791 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.013327 1815 raft_consensus.cc:2743] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:45.022691 1734 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:45.027081 1628 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:45.032436 1893 consensus_queue.cc:1035] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:45.028645 1815 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:45.034888 1815 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:45.035219 1815 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.035337 1838 consensus_queue.cc:1035] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20250114 02:53:45.035174 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:53:45.040293 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.051970 1894 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:45.055373 1628 leader_election.cc:343] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.061062 1779 tablet_bootstrap.cc:654] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.064668 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
W20250114 02:53:45.066555 1628 leader_election.cc:343] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.086104 1778 tablet_bootstrap.cc:654] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.088353 1894 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.100498 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:45.101086 1732 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.104444 1815 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.119125 1815 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.120957 1858 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.101110 1894 consensus_queue.cc:1035] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20250114 02:53:45.121040 1732 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:45.121721 1858 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.125157 1776 tablet_bootstrap.cc:654] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.125353 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:45.129630 1778 tablet_bootstrap.cc:492] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:45.130034 1778 ts_tablet_manager.cc:1397] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.156s user 0.018s sys 0.001s
W20250114 02:53:45.135510 1553 leader_election.cc:343] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:53:45.139285 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 48940.3us
I20250114 02:53:45.139525 1778 raft_consensus.cc:357] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.140412 1778 raft_consensus.cc:383] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.140652 1778 raft_consensus.cc:738] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.141350 1628 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:45.141191 1778 consensus_queue.cc:260] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.146656 1890 raft_consensus.cc:2798] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:45.146615 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:45.147125 1890 raft_consensus.cc:695] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:45.147087 1778 ts_tablet_manager.cc:1428] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20250114 02:53:45.147920 1890 consensus_queue.cc:237] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.149163 1898 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.149425 1778 tablet_bootstrap.cc:492] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
W20250114 02:53:45.148913 1628 leader_election.cc:343] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.147082 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:45.149621 1898 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.150847 1734 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.151906 1898 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.152241 1779 tablet_bootstrap.cc:492] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:45.152632 1779 ts_tablet_manager.cc:1397] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.163s user 0.024s sys 0.046s
I20250114 02:53:45.154616 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:53:45.155840 1553 leader_election.cc:343] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.156488 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.148597 1858 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:45.159301 1553 leader_election.cc:343] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.159807 1553 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:45.160616 1858 raft_consensus.cc:2743] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:45.162380 1734 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:45.165419 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.168825 1628 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:45.166126 1735 raft_consensus.cc:2463] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:45.173586 1894 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:45.182703 1553 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:45.185206 1858 raft_consensus.cc:2798] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:45.185518 1858 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:45.185824 1858 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.186836 1778 tablet_bootstrap.cc:654] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.188769 1485 catalog_manager.cc:5526] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:45.190452 1847 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.194636 1847 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.196334 1847 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:45.189922 1894 raft_consensus.cc:695] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:45.198206 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:45.198310 1779 raft_consensus.cc:357] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.199187 1779 raft_consensus.cc:383] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.199458 1779 raft_consensus.cc:738] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.200192 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.199514 1894 consensus_queue.cc:237] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:45.201543 1702 leader_election.cc:343] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:53:45.202184 1702 leader_election.cc:343] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.202464 1776 tablet_bootstrap.cc:492] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:45.203056 1776 ts_tablet_manager.cc:1397] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.228s user 0.017s sys 0.010s
I20250114 02:53:45.202613 1702 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:45.200361 1779 consensus_queue.cc:260] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.204239 1836 raft_consensus.cc:2743] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:45.205195 1779 ts_tablet_manager.cc:1428] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.052s user 0.005s sys 0.001s
I20250114 02:53:45.206426 1779 tablet_bootstrap.cc:492] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:45.204519 1858 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.210747 1776 raft_consensus.cc:357] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.211369 1776 raft_consensus.cc:383] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.211617 1776 raft_consensus.cc:738] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.212154 1776 consensus_queue.cc:260] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.212853 1858 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.214551 1776 ts_tablet_manager.cc:1428] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250114 02:53:45.215047 1779 tablet_bootstrap.cc:654] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.215610 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
W20250114 02:53:45.216827 1553 leader_election.cc:343] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.216816 1778 tablet_bootstrap.cc:492] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:45.217705 1778 ts_tablet_manager.cc:1397] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.004s
I20250114 02:53:45.218657 1776 tablet_bootstrap.cc:492] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:45.221581 1952 consensus_queue.cc:1035] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:45.224783 1779 tablet_bootstrap.cc:492] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:45.225189 1779 ts_tablet_manager.cc:1397] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.003s
I20250114 02:53:45.225200 1778 raft_consensus.cc:357] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.225936 1778 raft_consensus.cc:383] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.226217 1778 raft_consensus.cc:738] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.226251 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:45.226838 1735 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.227017 1778 consensus_queue.cc:260] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.227818 1779 raft_consensus.cc:357] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.228432 1779 raft_consensus.cc:383] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.228655 1779 raft_consensus.cc:738] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.229364 1779 consensus_queue.cc:260] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.231477 1779 ts_tablet_manager.cc:1428] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250114 02:53:45.232218 1779 tablet_bootstrap.cc:492] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:45.236826 1847 consensus_queue.cc:1035] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:45.237771 1779 tablet_bootstrap.cc:654] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.241319 1735 raft_consensus.cc:2463] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:45.242648 1778 ts_tablet_manager.cc:1428] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.025s user 0.003s sys 0.003s
I20250114 02:53:45.245357 1836 consensus_queue.cc:1035] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:45.248972 1776 tablet_bootstrap.cc:654] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.249841 1815 consensus_queue.cc:1035] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:45.256188 1778 tablet_bootstrap.cc:492] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:45.260890 1776 tablet_bootstrap.cc:492] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:45.261000 1553 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:45.261348 1776 ts_tablet_manager.cc:1397] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.043s user 0.016s sys 0.001s
I20250114 02:53:45.261688 1858 raft_consensus.cc:2798] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:45.262426 1858 raft_consensus.cc:695] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:45.263231 1858 consensus_queue.cc:237] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.266525 1776 raft_consensus.cc:357] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.267267 1776 raft_consensus.cc:383] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.267571 1815 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.268115 1815 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.269639 1778 tablet_bootstrap.cc:654] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.267582 1776 raft_consensus.cc:738] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.270628 1776 consensus_queue.cc:260] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.270529 1815 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.271939 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.272681 1734 raft_consensus.cc:2388] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:45.255260 1483 catalog_manager.cc:5526] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:45.273471 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.275051 1628 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:45.275527 1898 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.275913 1898 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.276417 1894 raft_consensus.cc:3054] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.282117 1894 raft_consensus.cc:2743] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:45.282621 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:45.285293 1898 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.285964 1776 ts_tablet_manager.cc:1428] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.024s user 0.006s sys 0.003s
I20250114 02:53:45.286404 1898 consensus_queue.cc:1035] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:45.287364 1776 tablet_bootstrap.cc:492] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:45.292616 1836 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.293025 1836 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.295096 1836 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:45.300707 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.303103 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:53:45.305569 1553 leader_election.cc:343] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.311832 1778 tablet_bootstrap.cc:492] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:45.312314 1778 ts_tablet_manager.cc:1397] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.056s user 0.017s sys 0.001s
I20250114 02:53:45.316049 1778 raft_consensus.cc:357] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.316787 1778 raft_consensus.cc:383] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.317095 1778 raft_consensus.cc:738] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.317688 1778 consensus_queue.cc:260] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.318068 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.318710 1734 raft_consensus.cc:2463] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:45.326198 1778 ts_tablet_manager.cc:1428] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.014s user 0.005s sys 0.003s
W20250114 02:53:45.326220 1702 leader_election.cc:343] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.327780 1776 tablet_bootstrap.cc:654] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
W20250114 02:53:45.328107 1702 consensus_peers.cc:487] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 -> Peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Couldn't send request to peer 1e5870f8269b49cf81dc39f1a5c8f5c1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250114 02:53:45.330632 1702 leader_election.cc:343] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.331318 1702 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:45.327431 1778 tablet_bootstrap.cc:492] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:45.337419 1836 raft_consensus.cc:2743] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:45.338407 1776 tablet_bootstrap.cc:492] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:45.338812 1776 ts_tablet_manager.cc:1397] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.052s user 0.014s sys 0.002s
I20250114 02:53:45.341074 1776 raft_consensus.cc:357] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.341670 1776 raft_consensus.cc:383] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.341895 1776 raft_consensus.cc:738] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.342916 1776 consensus_queue.cc:260] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.344918 1776 ts_tablet_manager.cc:1428] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20250114 02:53:45.345630 1776 tablet_bootstrap.cc:492] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:45.347748 1483 catalog_manager.cc:5526] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:45.350008 1553 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:45.350656 1779 tablet_bootstrap.cc:492] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:45.350800 1776 tablet_bootstrap.cc:654] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.351051 1779 ts_tablet_manager.cc:1397] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.119s user 0.020s sys 0.017s
I20250114 02:53:45.351423 1972 raft_consensus.cc:2798] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:45.351749 1972 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:45.351944 1972 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.353844 1779 raft_consensus.cc:357] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.357725 1972 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.360190 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:45.360909 1858 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.361683 1776 tablet_bootstrap.cc:492] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:45.361418 1858 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:45.361881 1553 leader_election.cc:343] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.362056 1776 ts_tablet_manager.cc:1397] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20250114 02:53:45.363241 1858 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.364534 1776 raft_consensus.cc:357] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.365355 1776 raft_consensus.cc:383] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.365439 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:45.365617 1776 raft_consensus.cc:738] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.366257 1776 consensus_queue.cc:260] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:45.366780 1553 leader_election.cc:343] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.368095 1972 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.368496 1776 ts_tablet_manager.cc:1428] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20250114 02:53:45.369408 1776 tablet_bootstrap.cc:492] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:45.354470 1779 raft_consensus.cc:383] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.374678 1779 raft_consensus.cc:738] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.375216 1779 consensus_queue.cc:260] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.377161 1779 ts_tablet_manager.cc:1428] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.026s user 0.003s sys 0.001s
I20250114 02:53:45.377732 1776 tablet_bootstrap.cc:654] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.380555 1779 tablet_bootstrap.cc:492] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:45.383343 1972 consensus_queue.cc:1035] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:45.385931 1779 tablet_bootstrap.cc:654] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.393836 1778 tablet_bootstrap.cc:654] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.400365 1776 tablet_bootstrap.cc:492] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:45.400801 1776 ts_tablet_manager.cc:1397] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.000s
I20250114 02:53:45.402258 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:45.403040 1734 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.402850 1776 raft_consensus.cc:357] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.403558 1776 raft_consensus.cc:383] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.403838 1776 raft_consensus.cc:738] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.404309 1779 tablet_bootstrap.cc:492] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:45.404687 1779 ts_tablet_manager.cc:1397] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.011s
I20250114 02:53:45.404412 1776 consensus_queue.cc:260] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.406353 1776 ts_tablet_manager.cc:1428] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250114 02:53:45.407224 1776 tablet_bootstrap.cc:492] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:45.406890 1779 raft_consensus.cc:357] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.407537 1779 raft_consensus.cc:383] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.407784 1779 raft_consensus.cc:738] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.408394 1779 consensus_queue.cc:260] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.410985 1779 ts_tablet_manager.cc:1428] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20250114 02:53:45.411198 1734 raft_consensus.cc:2463] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:45.411710 1779 tablet_bootstrap.cc:492] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:45.412612 1776 tablet_bootstrap.cc:654] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.413012 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.413760 1734 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:45.418350 1553 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:45.418722 1779 tablet_bootstrap.cc:654] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.420956 1972 raft_consensus.cc:2798] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:45.425228 1972 raft_consensus.cc:695] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:45.430521 1553 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:45.426426 1972 consensus_queue.cc:237] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.432051 1858 raft_consensus.cc:2798] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:45.432493 1858 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:45.432811 1858 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.439798 1858 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.441864 1858 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.443181 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:45.443763 1661 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.447891 1776 tablet_bootstrap.cc:492] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:45.448313 1776 ts_tablet_manager.cc:1397] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.016s
I20250114 02:53:45.450526 1776 raft_consensus.cc:357] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.451151 1776 raft_consensus.cc:383] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.451565 1776 raft_consensus.cc:738] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.452134 1776 consensus_queue.cc:260] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.453450 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:45.454335 1776 ts_tablet_manager.cc:1428] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250114 02:53:45.454668 1734 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.455704 1776 tablet_bootstrap.cc:492] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:45.451277 1661 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:45.460223 1734 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:45.460147 1553 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:45.461266 1858 raft_consensus.cc:2798] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:45.461771 1858 raft_consensus.cc:695] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:45.462687 1858 consensus_queue.cc:237] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.465786 1776 tablet_bootstrap.cc:654] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.477535 1839 consensus_queue.cc:1035] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:53:45.477711 1702 consensus_peers.cc:487] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 -> Peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Couldn't send request to peer 1e5870f8269b49cf81dc39f1a5c8f5c1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:45.480468 1776 tablet_bootstrap.cc:492] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:45.480763 1776 ts_tablet_manager.cc:1397] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.008s
I20250114 02:53:45.484648 1776 raft_consensus.cc:357] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.485196 1776 raft_consensus.cc:383] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.485421 1776 raft_consensus.cc:738] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.485890 1776 consensus_queue.cc:260] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.488909 1839 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.489153 1776 ts_tablet_manager.cc:1428] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250114 02:53:45.489464 1839 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.490893 1776 tablet_bootstrap.cc:492] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:45.490911 1839 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:45.497788 1776 tablet_bootstrap.cc:654] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.499509 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.501039 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:53:45.502553 1702 leader_election.cc:343] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:53:45.503893 1702 leader_election.cc:343] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.504341 1702 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:45.506310 1839 raft_consensus.cc:2743] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:45.513382 1778 tablet_bootstrap.cc:492] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:45.513752 1778 ts_tablet_manager.cc:1397] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.187s user 0.016s sys 0.001s
I20250114 02:53:45.515977 1778 raft_consensus.cc:357] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.516577 1778 raft_consensus.cc:383] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.516817 1778 raft_consensus.cc:738] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.517055 1483 catalog_manager.cc:5526] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:45.517371 1778 consensus_queue.cc:260] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.520965 1779 tablet_bootstrap.cc:492] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:45.521373 1779 ts_tablet_manager.cc:1397] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.110s user 0.028s sys 0.047s
I20250114 02:53:45.521979 1776 tablet_bootstrap.cc:492] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:45.522416 1776 ts_tablet_manager.cc:1397] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.011s
I20250114 02:53:45.523764 1779 raft_consensus.cc:357] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.524224 1952 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.524415 1779 raft_consensus.cc:383] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.524693 1779 raft_consensus.cc:738] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.524626 1952 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.524626 1776 raft_consensus.cc:357] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.525665 1776 raft_consensus.cc:383] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.525511 1779 consensus_queue.cc:260] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.525978 1776 raft_consensus.cc:738] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.526747 1776 consensus_queue.cc:260] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.527168 1952 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.528873 1776 ts_tablet_manager.cc:1428] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:53:45.529716 1776 tablet_bootstrap.cc:492] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:45.529618 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.530855 1779 ts_tablet_manager.cc:1428] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
W20250114 02:53:45.531096 1628 leader_election.cc:343] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.531584 1779 tablet_bootstrap.cc:492] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:45.532277 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:45.533478 1628 leader_election.cc:343] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.533887 1628 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:45.534667 1815 raft_consensus.cc:2743] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:45.537360 1894 consensus_queue.cc:1035] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:45.538632 1483 catalog_manager.cc:5526] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:45.539976 1776 tablet_bootstrap.cc:654] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.540517 1779 tablet_bootstrap.cc:654] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.542673 1839 consensus_queue.cc:1035] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:45.546175 1778 ts_tablet_manager.cc:1428] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20250114 02:53:45.546913 1778 tablet_bootstrap.cc:492] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:45.558411 1778 tablet_bootstrap.cc:654] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.568004 1779 tablet_bootstrap.cc:492] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:45.568384 1779 ts_tablet_manager.cc:1397] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.004s
I20250114 02:53:45.570983 1776 tablet_bootstrap.cc:492] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:45.571399 1776 ts_tablet_manager.cc:1397] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.015s
I20250114 02:53:45.571122 1779 raft_consensus.cc:357] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.571903 1779 raft_consensus.cc:383] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.572227 1779 raft_consensus.cc:738] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.572865 1779 consensus_queue.cc:260] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.574115 1776 raft_consensus.cc:357] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.574800 1776 raft_consensus.cc:383] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.575248 1779 ts_tablet_manager.cc:1428] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250114 02:53:45.575912 1778 tablet_bootstrap.cc:492] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:45.576081 1779 tablet_bootstrap.cc:492] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:45.576661 1778 ts_tablet_manager.cc:1397] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.001s
I20250114 02:53:45.575066 1776 raft_consensus.cc:738] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.577944 1776 consensus_queue.cc:260] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.581585 1776 ts_tablet_manager.cc:1428] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250114 02:53:45.581753 1778 raft_consensus.cc:357] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.582224 1776 tablet_bootstrap.cc:492] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:45.582345 1778 raft_consensus.cc:383] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.582623 1778 raft_consensus.cc:738] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.583199 1778 consensus_queue.cc:260] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.588454 1779 tablet_bootstrap.cc:654] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.589458 1778 ts_tablet_manager.cc:1428] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20250114 02:53:45.590272 1778 tablet_bootstrap.cc:492] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:45.592664 1776 tablet_bootstrap.cc:654] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.595772 1778 tablet_bootstrap.cc:654] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.600646 1585 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.605908 1779 tablet_bootstrap.cc:492] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:45.606381 1779 ts_tablet_manager.cc:1397] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.012s
I20250114 02:53:45.608542 1779 raft_consensus.cc:357] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.610379 1779 raft_consensus.cc:383] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.610718 1779 raft_consensus.cc:738] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.611639 1776 tablet_bootstrap.cc:492] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:45.612018 1776 ts_tablet_manager.cc:1397] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.000s
I20250114 02:53:45.612351 1958 consensus_queue.cc:1035] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:45.611469 1779 consensus_queue.cc:260] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.615782 1779 ts_tablet_manager.cc:1428] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.009s user 0.002s sys 0.002s
I20250114 02:53:45.616585 1779 tablet_bootstrap.cc:492] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:45.615451 1776 raft_consensus.cc:357] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.616916 1776 raft_consensus.cc:383] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.617148 1776 raft_consensus.cc:738] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.617636 1776 consensus_queue.cc:260] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.619612 1776 ts_tablet_manager.cc:1428] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 02:53:45.623219 1778 tablet_bootstrap.cc:492] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:45.623610 1778 ts_tablet_manager.cc:1397] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.005s
I20250114 02:53:45.624981 1779 tablet_bootstrap.cc:654] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.626370 1778 raft_consensus.cc:357] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.626993 1778 raft_consensus.cc:383] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.627259 1778 raft_consensus.cc:738] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.623525 1972 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.627899 1778 consensus_queue.cc:260] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.630090 1778 ts_tablet_manager.cc:1428] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:53:45.630391 1958 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.630692 1838 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.630901 1778 tablet_bootstrap.cc:492] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:45.630820 1958 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.631160 1838 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.632648 1958 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.632930 1838 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:45.621795 1776 tablet_bootstrap.cc:492] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:45.635003 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.635355 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.627964 1972 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.636317 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:45.636853 1661 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:45.637660 1972 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.637881 1702 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
W20250114 02:53:45.638655 1702 leader_election.cc:343] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.640352 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.640643 1836 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:45.640992 1836 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:45.641427 1836 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.641685 1776 tablet_bootstrap.cc:654] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
W20250114 02:53:45.643240 1628 leader_election.cc:343] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.644062 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.644659 1734 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:45.646212 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:45.646901 1661 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:45.647883 1553 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:45.650378 1972 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:45.650751 1972 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:45.651072 1972 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.651588 1778 tablet_bootstrap.cc:654] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.654171 1858 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.654651 1858 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.656414 1858 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.657986 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:45.660034 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.660773 1734 raft_consensus.cc:2388] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:45.663666 1778 tablet_bootstrap.cc:492] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:45.665792 1553 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:45.664230 1778 ts_tablet_manager.cc:1397] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.000s
I20250114 02:53:45.666718 1858 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.668010 1836 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.640964 1585 raft_consensus.cc:2463] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:45.669631 1778 raft_consensus.cc:357] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.670500 1778 raft_consensus.cc:383] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.670825 1778 raft_consensus.cc:738] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.671564 1778 consensus_queue.cc:260] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.672475 1836 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:45.672660 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:45.673192 1661 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.676889 1836 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.677158 1628 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:45.677412 1836 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.678149 1952 raft_consensus.cc:2798] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:45.678505 1952 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:45.678869 1952 raft_consensus.cc:3054] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.679739 1836 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:45.680750 1778 ts_tablet_manager.cc:1428] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250114 02:53:45.681910 1778 tablet_bootstrap.cc:492] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:45.683101 1893 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.683527 1893 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.684495 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:45.685149 1662 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:45.685317 1893 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.686072 1702 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:45.686194 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.687399 1838 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:45.687736 1838 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:45.688050 1838 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.688154 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.688730 1584 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:45.689874 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:45.684427 1952 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.694684 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
W20250114 02:53:45.695792 1628 leader_election.cc:343] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.696242 1952 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.696318 1778 tablet_bootstrap.cc:654] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.698546 1661 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:45.699606 1702 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:45.699837 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:45.700335 1847 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:45.700431 1586 raft_consensus.cc:3054] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.700811 1847 raft_consensus.cc:695] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:45.701627 1847 consensus_queue.cc:237] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.705623 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.707191 1586 raft_consensus.cc:2463] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:45.694332 1838 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.709497 1838 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:45.710078 1893 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.710633 1893 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.711777 1628 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:45.712734 1893 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.713304 1893 raft_consensus.cc:2743] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:45.713730 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:45.714279 1662 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.715574 1858 raft_consensus.cc:2743] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:45.717250 1778 tablet_bootstrap.cc:492] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:45.717778 1778 ts_tablet_manager.cc:1397] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.000s
I20250114 02:53:45.718937 1958 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.720083 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.720871 1736 raft_consensus.cc:2388] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:45.720854 1778 raft_consensus.cc:357] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.719384 1958 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.717281 1972 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.723351 1958 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:45.725502 1702 leader_election.cc:343] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.727996 1485 catalog_manager.cc:5526] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:45.729370 1778 raft_consensus.cc:383] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.726701 1628 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:45.727335 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:45.729758 1778 raft_consensus.cc:738] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.731308 1894 raft_consensus.cc:2798] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:45.733937 1778 consensus_queue.cc:260] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.736491 1778 ts_tablet_manager.cc:1428] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20250114 02:53:45.737311 1973 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.737702 1973 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.738418 1894 raft_consensus.cc:695] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:45.738860 1778 tablet_bootstrap.cc:492] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:45.739240 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.739581 1973 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.739320 1894 consensus_queue.cc:237] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.741158 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.742733 1736 raft_consensus.cc:2388] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:45.734130 1958 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.744678 1958 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.757867 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.742138 1585 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.762399 1628 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:45.761754 1952 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.760679 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:45.765028 1734 raft_consensus.cc:2463] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:45.760838 1815 consensus_queue.cc:1035] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250114 02:53:45.771453 1972 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.776259 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:45.776752 1735 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.767815 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.780097 1553 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:45.780710 1779 tablet_bootstrap.cc:492] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:45.781062 1779 ts_tablet_manager.cc:1397] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.165s user 0.039s sys 0.061s
I20250114 02:53:45.782227 1776 tablet_bootstrap.cc:492] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:45.782533 1776 ts_tablet_manager.cc:1397] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.161s user 0.027s sys 0.038s
I20250114 02:53:45.784818 1776 raft_consensus.cc:357] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.784796 2002 consensus_queue.cc:1035] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:45.785554 1776 raft_consensus.cc:383] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.785809 1776 raft_consensus.cc:738] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.774020 1836 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.778713 1893 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.777343 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:45.792169 1776 consensus_queue.cc:260] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.799460 1779 raft_consensus.cc:357] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.800094 1779 raft_consensus.cc:383] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:45.800272 1778 tablet_bootstrap.cc:654] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:45.800359 1779 raft_consensus.cc:738] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:45.773149 1952 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.804425 1779 consensus_queue.cc:260] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.786712 1972 raft_consensus.cc:2798] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:45.806761 1972 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:45.807166 1972 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.807418 1779 ts_tablet_manager.cc:1428] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.026s user 0.004s sys 0.003s
I20250114 02:53:45.808221 1779 tablet_bootstrap.cc:492] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
W20250114 02:53:45.817739 1553 leader_election.cc:343] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.819427 1661 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.819509 1972 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.821295 1972 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.821990 1483 catalog_manager.cc:5526] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:45.825233 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:45.825865 1736 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.826359 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:45.831296 1898 consensus_queue.cc:1035] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:45.836212 1736 raft_consensus.cc:2463] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:45.837389 1553 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:45.838220 1898 raft_consensus.cc:2798] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:45.838821 1898 raft_consensus.cc:695] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:45.839820 1898 consensus_queue.cc:237] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.805167 1660 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.846539 1628 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:45.791966 1836 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.851608 1776 ts_tablet_manager.cc:1428] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.069s user 0.007s sys 0.000s
W20250114 02:53:45.853061 1553 leader_election.cc:343] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:45.855039 1776 tablet_bootstrap.cc:492] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:45.852957 1958 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.858700 1973 consensus_queue.cc:1035] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250114 02:53:45.866050 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.867866 1736 raft_consensus.cc:2463] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:45.870314 1972 consensus_queue.cc:1035] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:45.873051 1836 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:45.874821 1628 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:45.881114 1894 raft_consensus.cc:2798] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:45.881500 1894 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:45.858718 1952 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.861200 1662 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:45.881778 1894 raft_consensus.cc:3054] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.924978 1894 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.937274 1702 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:45.942848 1839 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:45.952898 1839 raft_consensus.cc:695] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:45.953640 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:45.953823 1839 consensus_queue.cc:237] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.954540 1733 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:45.964380 1972 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:45.983197 1972 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:45.984257 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:45.984591 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.985071 1972 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.947660 1894 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:45.961194 2004 raft_consensus.cc:2743] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:45.989265 1628 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:45.963510 1657 raft_consensus.cc:3054] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.989424 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:45.993861 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:45.994072 1731 raft_consensus.cc:3054] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:45.998705 1584 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:45.999948 1735 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:45.960132 1973 consensus_queue.cc:1035] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20250114 02:53:45.988483 1893 raft_consensus.cc:2743] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:45.992656 1894 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:46.014394 1778 tablet_bootstrap.cc:492] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:46.015169 1778 ts_tablet_manager.cc:1397] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.277s user 0.027s sys 0.041s
W20250114 02:53:46.016047 1628 leader_election.cc:343] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:53:46.017433 1628 leader_election.cc:343] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:46.021636 1779 tablet_bootstrap.cc:654] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.021292 1553 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:46.023006 1858 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:46.023451 1858 raft_consensus.cc:695] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:46.023646 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:46.018298 1894 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:46.009843 1585 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:46.033372 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:46.033959 1735 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:46.052521 1484 catalog_manager.cc:5526] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
W20250114 02:53:46.055966 1702 leader_election.cc:343] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:46.058995 1553 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:46.040733 1894 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.024228 1858 consensus_queue.cc:237] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.054375 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:46.063746 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:46.066226 1778 raft_consensus.cc:357] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.066887 1778 raft_consensus.cc:383] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.067147 1778 raft_consensus.cc:738] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.067684 1778 consensus_queue.cc:260] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.069186 1778 ts_tablet_manager.cc:1428] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20250114 02:53:46.069922 1778 tablet_bootstrap.cc:492] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:46.061687 1898 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:46.076754 1898 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 02:53:46.088768 1702 leader_election.cc:343] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:46.089640 1702 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:46.090931 1836 raft_consensus.cc:2743] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:46.085497 1660 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:46.089529 1898 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.098254 1972 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:46.116624 1776 tablet_bootstrap.cc:654] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
W20250114 02:53:46.114805 2011 log.cc:926] Time spent T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20250114 02:53:46.119750 1731 raft_consensus.cc:2463] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:46.121296 1628 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:46.132745 2005 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:46.133340 2005 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.110632 1972 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.129966 2002 raft_consensus.cc:2798] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:46.142998 2002 raft_consensus.cc:695] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:46.151944 1972 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:46.161444 2009 log.cc:926] Time spent T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20250114 02:53:46.159106 1972 consensus_queue.cc:1035] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250114 02:53:46.164783 1972 consensus_queue.cc:1035] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:46.165668 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:46.171840 1779 tablet_bootstrap.cc:492] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:46.143827 2002 consensus_queue.cc:237] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.172173 1779 ts_tablet_manager.cc:1397] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.364s user 0.018s sys 0.019s
I20250114 02:53:46.171934 2005 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:46.178704 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:46.180888 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:46.188087 1779 raft_consensus.cc:357] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.188872 1779 raft_consensus.cc:383] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.189230 1779 raft_consensus.cc:738] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.191615 1898 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.189977 1779 consensus_queue.cc:260] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:46.194411 1628 leader_election.cc:343] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:46.204665 1778 tablet_bootstrap.cc:654] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
W20250114 02:53:46.209028 1628 consensus_peers.cc:487] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 -> Peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Couldn't send request to peer b89550620e1d455d887be72232bb5454. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:46.215595 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:46.212769 1660 raft_consensus.cc:2388] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:46.224853 1779 ts_tablet_manager.cc:1428] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.052s user 0.007s sys 0.000s
I20250114 02:53:46.191638 1894 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.235041 1776 tablet_bootstrap.cc:492] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:46.235373 1776 ts_tablet_manager.cc:1397] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.380s user 0.022s sys 0.010s
I20250114 02:53:46.237840 1779 tablet_bootstrap.cc:492] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:46.242223 1553 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:46.246359 1661 raft_consensus.cc:2388] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:46.243830 1858 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.250906 1484 catalog_manager.cc:5526] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
W20250114 02:53:46.259349 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20250114 02:53:46.266638 2015 log.cc:926] Time spent T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20250114 02:53:46.264945 1898 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:46.270879 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:46.284197 1776 raft_consensus.cc:357] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.286643 1776 raft_consensus.cc:383] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.287354 1776 raft_consensus.cc:738] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.287750 1776 consensus_queue.cc:260] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.277828 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:46.295634 1660 raft_consensus.cc:2388] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:46.295820 1836 consensus_queue.cc:1035] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:46.288749 1732 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.300462 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:46.301136 1586 raft_consensus.cc:2388] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
W20250114 02:53:46.302928 2016 log.cc:926] Time spent T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20250114 02:53:46.305356 1778 tablet_bootstrap.cc:492] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:46.305754 1778 ts_tablet_manager.cc:1397] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.236s user 0.021s sys 0.013s
W20250114 02:53:46.308073 1553 consensus_peers.cc:487] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd -> Peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Couldn't send request to peer 1e5870f8269b49cf81dc39f1a5c8f5c1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:46.309538 1776 ts_tablet_manager.cc:1428] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.074s user 0.006s sys 0.000s
I20250114 02:53:46.309139 1483 catalog_manager.cc:5526] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:46.310377 1776 tablet_bootstrap.cc:492] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:46.310817 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:46.314831 1661 raft_consensus.cc:3054] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:53:46.319347 1628 leader_election.cc:343] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:46.320762 1628 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:46.302075 1894 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:46.330689 1778 raft_consensus.cc:357] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.331513 1778 raft_consensus.cc:383] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.331825 1778 raft_consensus.cc:738] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.332515 1778 consensus_queue.cc:260] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.324677 2004 raft_consensus.cc:2743] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:53:46.348578 2023 log.cc:926] Time spent T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20250114 02:53:46.349057 1778 ts_tablet_manager.cc:1428] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.043s user 0.006s sys 0.001s
I20250114 02:53:46.337430 1958 consensus_queue.cc:1035] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250114 02:53:46.349934 1778 tablet_bootstrap.cc:492] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:46.352141 1776 tablet_bootstrap.cc:654] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.356822 1972 consensus_queue.cc:1035] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:46.365109 1778 tablet_bootstrap.cc:654] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.369263 1628 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:46.359773 1858 raft_consensus.cc:2743] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:46.379298 1893 raft_consensus.cc:2743] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 02:53:46.379043 1847 consensus_queue.cc:1035] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250114 02:53:46.425971 1779 tablet_bootstrap.cc:654] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.428421 1732 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:46.458961 1553 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:46.460240 1898 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:46.460767 1898 raft_consensus.cc:695] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:46.461697 1898 consensus_queue.cc:237] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.468771 1847 consensus_queue.cc:1035] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:46.506614 1858 consensus_queue.cc:1035] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:46.510701 1776 tablet_bootstrap.cc:492] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:46.511161 1776 ts_tablet_manager.cc:1397] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.201s user 0.012s sys 0.003s
I20250114 02:53:46.513764 1776 raft_consensus.cc:357] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.514494 1776 raft_consensus.cc:383] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.516767 1776 raft_consensus.cc:738] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.517676 1776 consensus_queue.cc:260] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.523224 1778 tablet_bootstrap.cc:492] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:46.523656 1778 ts_tablet_manager.cc:1397] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.174s user 0.007s sys 0.006s
I20250114 02:53:46.538837 1483 catalog_manager.cc:5526] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:46.541138 1847 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:46.541558 1847 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.543712 1847 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:46.547163 1776 ts_tablet_manager.cc:1428] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20250114 02:53:46.548136 1776 tablet_bootstrap.cc:492] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:46.552103 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:46.549558 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:46.553158 1661 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
W20250114 02:53:46.553877 1702 leader_election.cc:343] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:46.554641 1702 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:46.555681 1839 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:46.555992 1839 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:46.556301 1839 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.561724 1776 tablet_bootstrap.cc:654] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.561013 2012 consensus_queue.cc:1035] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:46.564286 1893 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:46.564692 1893 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.566392 1893 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:46.569173 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:46.569072 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:53:46.571108 1628 leader_election.cc:343] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:53:46.571712 1628 leader_election.cc:343] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:46.572053 1628 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:46.572919 1893 raft_consensus.cc:2743] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:46.573153 1839 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.577253 1779 tablet_bootstrap.cc:492] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:46.577646 1779 ts_tablet_manager.cc:1397] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.341s user 0.021s sys 0.014s
I20250114 02:53:46.578603 1778 raft_consensus.cc:357] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.579277 1778 raft_consensus.cc:383] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.579538 1778 raft_consensus.cc:738] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.577704 1838 consensus_queue.cc:1035] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:46.580984 1778 consensus_queue.cc:260] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.584031 1839 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:46.585177 1778 ts_tablet_manager.cc:1428] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.061s user 0.005s sys 0.001s
I20250114 02:53:46.585908 1778 tablet_bootstrap.cc:492] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:46.585940 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:46.586247 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:46.586582 1660 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:53:46.587235 1702 leader_election.cc:343] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:46.588027 1485 catalog_manager.cc:5526] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:46.597443 2012 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:46.598012 2012 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.588404 1779 raft_consensus.cc:357] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.602794 1779 raft_consensus.cc:383] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.603142 1779 raft_consensus.cc:738] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.609777 1779 consensus_queue.cc:260] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.616706 1779 ts_tablet_manager.cc:1428] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.039s user 0.006s sys 0.003s
I20250114 02:53:46.616847 2012 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:46.617712 1779 tablet_bootstrap.cc:492] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:46.619313 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:53:46.620555 1553 leader_election.cc:343] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:46.620746 1660 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:46.621867 1778 tablet_bootstrap.cc:654] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.622784 1702 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:46.625061 1839 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:46.625499 1839 raft_consensus.cc:695] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:46.626257 1839 consensus_queue.cc:237] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.630650 1776 tablet_bootstrap.cc:492] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:46.631052 1776 ts_tablet_manager.cc:1397] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.083s user 0.013s sys 0.000s
I20250114 02:53:46.636325 1776 raft_consensus.cc:357] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.636966 1776 raft_consensus.cc:383] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.637214 1776 raft_consensus.cc:738] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.637765 1776 consensus_queue.cc:260] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.640177 1776 ts_tablet_manager.cc:1428] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250114 02:53:46.642056 1776 tablet_bootstrap.cc:492] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:46.639822 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:53:46.643057 1553 leader_election.cc:343] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:46.643429 1553 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:46.644312 1779 tablet_bootstrap.cc:654] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.644435 2012 raft_consensus.cc:2743] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:46.647485 1898 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:46.647945 1898 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.651052 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:46.651669 1778 tablet_bootstrap.cc:492] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:46.652060 1778 ts_tablet_manager.cc:1397] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.002s
I20250114 02:53:46.654651 1898 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:46.655232 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:46.655022 1485 catalog_manager.cc:5526] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:46.655947 1734 raft_consensus.cc:2388] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:46.656747 1776 tablet_bootstrap.cc:654] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.657064 1553 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:46.660377 1778 raft_consensus.cc:357] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.661125 1778 raft_consensus.cc:383] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.661424 1778 raft_consensus.cc:738] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.662209 1778 consensus_queue.cc:260] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.663429 2012 raft_consensus.cc:3054] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.676502 1776 tablet_bootstrap.cc:492] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:46.676934 1776 ts_tablet_manager.cc:1397] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.002s
I20250114 02:53:46.678200 1778 ts_tablet_manager.cc:1428] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.026s user 0.005s sys 0.002s
I20250114 02:53:46.679317 1776 raft_consensus.cc:357] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.680001 1776 raft_consensus.cc:383] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.680289 1776 raft_consensus.cc:738] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.680544 1779 tablet_bootstrap.cc:492] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:46.681046 1779 ts_tablet_manager.cc:1397] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.064s user 0.011s sys 0.007s
I20250114 02:53:46.680889 1776 consensus_queue.cc:260] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.683346 1776 ts_tablet_manager.cc:1428] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20250114 02:53:46.683372 1779 raft_consensus.cc:357] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.684139 1779 raft_consensus.cc:383] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.684309 1776 tablet_bootstrap.cc:492] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:46.684399 1779 raft_consensus.cc:738] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.685060 1779 consensus_queue.cc:260] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.686470 2012 raft_consensus.cc:2743] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:46.679107 1778 tablet_bootstrap.cc:492] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:46.689633 1776 tablet_bootstrap.cc:654] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.695650 1778 tablet_bootstrap.cc:654] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.701793 1776 tablet_bootstrap.cc:492] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:46.700507 1779 ts_tablet_manager.cc:1428] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20250114 02:53:46.702163 1776 ts_tablet_manager.cc:1397] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.013s
I20250114 02:53:46.702884 1779 tablet_bootstrap.cc:492] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:46.704756 1776 raft_consensus.cc:357] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.705191 1776 raft_consensus.cc:383] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.705375 1776 raft_consensus.cc:738] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.705891 1776 consensus_queue.cc:260] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.707985 1776 ts_tablet_manager.cc:1428] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 02:53:46.708679 1776 tablet_bootstrap.cc:492] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:46.711566 1779 tablet_bootstrap.cc:654] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.713727 1776 tablet_bootstrap.cc:654] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.715014 1839 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:46.715482 1778 tablet_bootstrap.cc:492] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:46.715452 1839 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.715999 1778 ts_tablet_manager.cc:1397] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.000s
I20250114 02:53:46.719763 1778 raft_consensus.cc:357] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.720466 1778 raft_consensus.cc:383] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.720757 1778 raft_consensus.cc:738] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.721352 1778 consensus_queue.cc:260] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.723373 1778 ts_tablet_manager.cc:1428] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250114 02:53:46.724457 1778 tablet_bootstrap.cc:492] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:46.725920 1731 raft_consensus.cc:3054] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.734264 1839 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:46.734211 1661 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.735721 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:46.734258 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:46.737092 1778 tablet_bootstrap.cc:654] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.738862 1702 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:46.749673 1776 tablet_bootstrap.cc:492] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:46.750304 1776 ts_tablet_manager.cc:1397] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.009s
I20250114 02:53:46.753872 1776 raft_consensus.cc:357] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.754640 1776 raft_consensus.cc:383] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.754925 1776 raft_consensus.cc:738] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.755506 1776 consensus_queue.cc:260] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.757414 1776 ts_tablet_manager.cc:1428] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250114 02:53:46.759548 1776 tablet_bootstrap.cc:492] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:46.763123 1778 tablet_bootstrap.cc:492] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:46.763593 1778 ts_tablet_manager.cc:1397] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.005s
I20250114 02:53:46.768666 1776 tablet_bootstrap.cc:654] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.769970 1839 raft_consensus.cc:2743] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:46.766376 1778 raft_consensus.cc:357] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.772226 1778 raft_consensus.cc:383] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.772505 1778 raft_consensus.cc:738] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.773051 1778 consensus_queue.cc:260] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.773917 1779 tablet_bootstrap.cc:492] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:46.774379 1779 ts_tablet_manager.cc:1397] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.000s
I20250114 02:53:46.775019 1778 ts_tablet_manager.cc:1428] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250114 02:53:46.775862 1778 tablet_bootstrap.cc:492] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:46.777309 1779 raft_consensus.cc:357] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.777989 1779 raft_consensus.cc:383] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.778293 1779 raft_consensus.cc:738] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.778993 1779 consensus_queue.cc:260] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.781049 1778 tablet_bootstrap.cc:654] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.781590 1779 ts_tablet_manager.cc:1428] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250114 02:53:46.782351 1779 tablet_bootstrap.cc:492] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:46.785650 1776 tablet_bootstrap.cc:492] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:46.786085 1776 ts_tablet_manager.cc:1397] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.005s
I20250114 02:53:46.787889 1893 consensus_queue.cc:1035] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:46.790890 1776 raft_consensus.cc:357] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.791725 1776 raft_consensus.cc:383] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.792063 1776 raft_consensus.cc:738] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.792774 1776 consensus_queue.cc:260] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.795682 1776 ts_tablet_manager.cc:1428] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.009s user 0.000s sys 0.006s
I20250114 02:53:46.796543 1776 tablet_bootstrap.cc:492] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:46.804133 1778 tablet_bootstrap.cc:492] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:46.804500 1776 tablet_bootstrap.cc:654] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.804530 1778 ts_tablet_manager.cc:1397] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.012s
I20250114 02:53:46.808153 1779 tablet_bootstrap.cc:654] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.812669 1778 raft_consensus.cc:357] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.814247 1778 raft_consensus.cc:383] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.814456 1778 raft_consensus.cc:738] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.814973 1778 consensus_queue.cc:260] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.816872 1778 ts_tablet_manager.cc:1428] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20250114 02:53:46.817621 1778 tablet_bootstrap.cc:492] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:46.822856 1778 tablet_bootstrap.cc:654] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.830086 1894 consensus_queue.cc:1035] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:46.844285 1776 tablet_bootstrap.cc:492] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:46.845911 1776 ts_tablet_manager.cc:1397] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.001s
I20250114 02:53:46.848090 1776 raft_consensus.cc:357] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.848812 1776 raft_consensus.cc:383] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.854146 1776 raft_consensus.cc:738] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.854782 1776 consensus_queue.cc:260] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.856751 1776 ts_tablet_manager.cc:1428] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250114 02:53:46.858415 1779 tablet_bootstrap.cc:492] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:46.858860 1779 ts_tablet_manager.cc:1397] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.077s user 0.029s sys 0.009s
I20250114 02:53:46.859526 1776 tablet_bootstrap.cc:492] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:46.858939 1778 tablet_bootstrap.cc:492] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:46.862874 1778 ts_tablet_manager.cc:1397] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.045s user 0.020s sys 0.017s
I20250114 02:53:46.864416 1779 raft_consensus.cc:357] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.865072 1779 raft_consensus.cc:383] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.865428 1779 raft_consensus.cc:738] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.865149 1778 raft_consensus.cc:357] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.865859 1778 raft_consensus.cc:383] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.866170 1778 raft_consensus.cc:738] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.873167 1778 consensus_queue.cc:260] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.873497 1779 consensus_queue.cc:260] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.874501 1776 tablet_bootstrap.cc:654] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.881853 1778 ts_tablet_manager.cc:1428] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20250114 02:53:46.882612 1778 tablet_bootstrap.cc:492] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:46.882915 1779 ts_tablet_manager.cc:1428] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.021s user 0.006s sys 0.004s
I20250114 02:53:46.883924 1779 tablet_bootstrap.cc:492] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:46.886873 1776 tablet_bootstrap.cc:492] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:46.887240 1776 ts_tablet_manager.cc:1397] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.006s
I20250114 02:53:46.889645 1776 raft_consensus.cc:357] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.890291 1776 raft_consensus.cc:383] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.890530 1776 raft_consensus.cc:738] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.890709 1779 tablet_bootstrap.cc:654] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.890719 2012 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:46.892164 1778 tablet_bootstrap.cc:654] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.891758 1776 consensus_queue.cc:260] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.891829 2012 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.893626 2012 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:46.894037 1776 ts_tablet_manager.cc:1428] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250114 02:53:46.895088 1776 tablet_bootstrap.cc:492] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:46.896200 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:46.896901 1661 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:46.897987 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:53:46.899776 1553 leader_election.cc:343] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:46.900521 1553 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:46.901682 1973 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:46.902077 1973 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:46.902485 1973 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.905540 2012 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:46.911151 2012 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.918905 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:46.919693 1661 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:46.920697 1553 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:46.921406 1898 raft_consensus.cc:2798] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:46.921768 1898 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:46.922068 1898 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.925254 1972 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:46.925787 1972 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.927294 1858 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:46.927953 1972 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:46.927824 1858 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.930105 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:46.932020 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:46.932415 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:46.932941 1660 raft_consensus.cc:2388] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
W20250114 02:53:46.933876 1553 leader_election.cc:343] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:46.934183 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:46.917773 2012 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:46.938715 1858 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:46.939200 1731 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:46.937248 1553 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:46.944360 2012 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.944600 1973 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.946331 1973 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:46.945111 1776 tablet_bootstrap.cc:654] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.946933 1778 tablet_bootstrap.cc:492] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:46.947304 1778 ts_tablet_manager.cc:1397] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.065s user 0.024s sys 0.023s
I20250114 02:53:46.949687 1778 raft_consensus.cc:357] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.950516 1778 raft_consensus.cc:383] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.950834 1778 raft_consensus.cc:738] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.951541 1778 consensus_queue.cc:260] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.954155 1778 ts_tablet_manager.cc:1428] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20250114 02:53:46.956591 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:46.957077 1779 tablet_bootstrap.cc:492] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:46.957443 1779 ts_tablet_manager.cc:1397] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.074s user 0.021s sys 0.030s
I20250114 02:53:46.959270 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:46.960100 1779 raft_consensus.cc:357] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.960757 1779 raft_consensus.cc:383] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:46.961009 1779 raft_consensus.cc:738] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:46.959975 1661 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.962550 1973 consensus_queue.cc:1035] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:46.942971 1898 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.950522 2012 raft_consensus.cc:2743] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:46.966279 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:46.966917 1662 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.970288 1661 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:46.964432 1898 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:46.973555 1778 tablet_bootstrap.cc:492] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:46.973403 1662 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:46.975059 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
W20250114 02:53:46.975248 1553 leader_election.cc:343] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:46.975608 1553 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:46.976274 2012 raft_consensus.cc:2743] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:46.978296 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:46.979446 1731 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:46.980628 1553 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:46.981882 1778 tablet_bootstrap.cc:654] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:46.982636 1973 raft_consensus.cc:2798] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:46.983135 1973 raft_consensus.cc:695] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:46.983780 1973 consensus_queue.cc:237] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.961534 1779 consensus_queue.cc:260] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:46.991286 1779 ts_tablet_manager.cc:1428] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.034s user 0.005s sys 0.001s
I20250114 02:53:46.997689 1779 tablet_bootstrap.cc:492] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:46.999287 1838 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.001236 1838 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.003289 1778 tablet_bootstrap.cc:492] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:46.987393 1731 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:46.999718 1483 catalog_manager.cc:5526] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:47.007318 1779 tablet_bootstrap.cc:654] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:47.003612 1778 ts_tablet_manager.cc:1397] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.011s
I20250114 02:53:47.003032 1838 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:47.005648 1553 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:47.018584 1973 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:47.019133 1973 raft_consensus.cc:695] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:47.019045 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:47.020252 1894 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.019982 1973 consensus_queue.cc:237] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.020666 1894 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.022433 1894 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:47.024435 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:47.025279 1553 leader_election.cc:343] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:47.025410 1585 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:47.011010 1778 raft_consensus.cc:357] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.026480 1628 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:47.026702 1778 raft_consensus.cc:383] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:47.026973 1778 raft_consensus.cc:738] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:47.027375 1894 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:47.027776 1894 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:47.028188 1894 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:47.028576 1858 consensus_queue.cc:1035] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:47.028956 1778 consensus_queue.cc:260] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.037824 1778 ts_tablet_manager.cc:1428] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20250114 02:53:47.038597 1778 tablet_bootstrap.cc:492] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:47.043054 1779 tablet_bootstrap.cc:492] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:47.042837 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:47.044735 1778 tablet_bootstrap.cc:654] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:47.045830 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:47.051896 1657 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:47.052486 1731 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:47.053045 1779 ts_tablet_manager.cc:1397] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.055s user 0.013s sys 0.002s
I20250114 02:53:47.049975 1776 tablet_bootstrap.cc:492] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:47.053876 1776 ts_tablet_manager.cc:1397] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.159s user 0.024s sys 0.021s
I20250114 02:53:47.049805 1894 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.056823 1779 raft_consensus.cc:357] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:47.051192 1702 leader_election.cc:343] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:47.057646 1779 raft_consensus.cc:383] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:47.058019 1779 raft_consensus.cc:738] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:47.059450 1779 consensus_queue.cc:260] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.060880 1776 raft_consensus.cc:357] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.061450 1776 raft_consensus.cc:383] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:47.061678 1776 raft_consensus.cc:738] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:47.062101 1779 ts_tablet_manager.cc:1428] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250114 02:53:47.063405 1779 tablet_bootstrap.cc:492] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:47.063560 1485 catalog_manager.cc:5526] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:47.062176 1776 consensus_queue.cc:260] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.067911 1702 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:47.069730 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:47.070688 1734 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:47.071727 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:47.072077 1779 tablet_bootstrap.cc:654] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:47.072286 1586 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:47.074888 1836 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.075311 1836 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.076977 1836 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:47.070199 1894 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:47.081525 1836 raft_consensus.cc:2798] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:47.081828 1836 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:47.082108 1836 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:47.085208 1734 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:47.086433 1628 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:47.087237 2005 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.087507 1776 ts_tablet_manager.cc:1428] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.033s user 0.003s sys 0.003s
I20250114 02:53:47.087611 2005 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.089313 2005 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:47.089731 2005 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:47.090157 2005 raft_consensus.cc:695] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:47.092139 1586 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:47.092351 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:47.093492 1776 tablet_bootstrap.cc:492] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
W20250114 02:53:47.094427 1628 leader_election.cc:343] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:47.093302 2005 consensus_queue.cc:237] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.101603 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:47.106029 1734 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:47.113232 1628 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:47.115705 1893 raft_consensus.cc:2798] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:47.116005 1893 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:47.116297 1893 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:47.101079 1836 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.130640 1776 tablet_bootstrap.cc:654] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
W20250114 02:53:47.112188 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 44355.5us
I20250114 02:53:47.121877 1893 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.135169 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:47.137367 1836 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:47.138267 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:47.138736 1958 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.146646 1838 consensus_queue.cc:1035] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:47.147673 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:47.139833 1585 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:47.149286 1893 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:47.150462 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:47.151160 1731 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:47.151099 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:47.151650 1779 tablet_bootstrap.cc:492] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
W20250114 02:53:47.152222 1702 leader_election.cc:343] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:47.152184 1779 ts_tablet_manager.cc:1397] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.089s user 0.021s sys 0.054s
I20250114 02:53:47.151669 1658 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:47.150053 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:47.160303 1779 raft_consensus.cc:357] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.161090 1779 raft_consensus.cc:383] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:47.161383 1779 raft_consensus.cc:738] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:47.162056 1779 consensus_queue.cc:260] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:47.166863 1628 leader_election.cc:343] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:47.140075 1958 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:47.173904 1702 leader_election.cc:343] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:47.174396 2012 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.174500 1702 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:47.174887 2012 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.175925 1839 raft_consensus.cc:2798] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:47.176245 1839 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:47.176553 1839 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:47.186077 1779 ts_tablet_manager.cc:1428] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
I20250114 02:53:47.186956 1779 tablet_bootstrap.cc:492] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:47.188503 1778 tablet_bootstrap.cc:492] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:47.188269 1658 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:47.188916 1778 ts_tablet_manager.cc:1397] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.151s user 0.037s sys 0.049s
I20250114 02:53:47.192365 1778 raft_consensus.cc:357] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.193558 1778 raft_consensus.cc:383] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:47.193872 1778 raft_consensus.cc:738] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:47.195559 2012 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:47.196864 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:47.197456 1658 raft_consensus.cc:2463] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:47.212092 1553 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:47.212236 1778 consensus_queue.cc:260] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.214828 1702 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:47.215058 1485 catalog_manager.cc:5526] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:47.216993 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:47.218070 1730 raft_consensus.cc:2463] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:47.214277 2012 raft_consensus.cc:2798] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:47.219136 2012 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:47.219753 2012 raft_consensus.cc:3054] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:47.219780 1958 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:47.220587 1838 raft_consensus.cc:2798] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:47.221019 1838 raft_consensus.cc:695] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:47.221949 1838 consensus_queue.cc:237] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.225821 2012 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.226138 1839 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.218415 1731 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:47.228511 1839 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:47.231109 1628 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:47.242667 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:47.243310 1731 raft_consensus.cc:2463] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:47.238967 2005 raft_consensus.cc:2798] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:47.243692 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:47.245901 2005 raft_consensus.cc:695] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:47.246246 1585 raft_consensus.cc:2388] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:47.244717 1628 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:47.246976 1958 raft_consensus.cc:2798] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:47.246692 2005 consensus_queue.cc:237] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.247278 1958 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:47.247591 1958 raft_consensus.cc:3054] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:47.251868 1778 ts_tablet_manager.cc:1428] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.063s user 0.006s sys 0.009s
I20250114 02:53:47.253065 1778 tablet_bootstrap.cc:492] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:47.255813 2012 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:47.256385 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:47.256717 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:47.257215 1730 raft_consensus.cc:3054] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:47.258248 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:47.260439 1587 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:47.229980 2079 consensus_queue.cc:1035] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20250114 02:53:47.263756 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:47.250061 1839 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.277185 1779 tablet_bootstrap.cc:654] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:47.267256 2081 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.279696 2081 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.280553 1776 tablet_bootstrap.cc:492] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:47.280942 1776 ts_tablet_manager.cc:1397] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.188s user 0.020s sys 0.011s
I20250114 02:53:47.281080 1778 tablet_bootstrap.cc:654] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:47.284466 1776 raft_consensus.cc:357] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.284744 1958 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.285053 1776 raft_consensus.cc:383] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:47.286716 1776 raft_consensus.cc:738] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:47.288336 1660 raft_consensus.cc:2388] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:47.289981 1776 consensus_queue.cc:260] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.291937 1776 ts_tablet_manager.cc:1428] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250114 02:53:47.292619 1776 tablet_bootstrap.cc:492] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:47.293012 1730 raft_consensus.cc:2463] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:47.295238 1553 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
W20250114 02:53:47.295547 1702 leader_election.cc:343] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:47.290486 1587 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:47.302062 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:47.297153 2012 raft_consensus.cc:2798] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:47.298771 1958 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:47.279142 1839 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.314806 1702 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:47.313557 2012 raft_consensus.cc:695] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:47.320854 2012 consensus_queue.cc:237] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.335637 1839 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:47.737018 1779 tablet_bootstrap.cc:492] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:47.322090 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:47.742774 1779 ts_tablet_manager.cc:1397] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.556s user 0.020s sys 0.016s
I20250114 02:53:47.743141 1730 raft_consensus.cc:2388] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:47.325909 1484 catalog_manager.cc:5526] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:47.332018 1485 catalog_manager.cc:5526] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:47.415324 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:47.412305 2081 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:47.415692 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:47.400487 1858 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.756883 1858 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.759243 1858 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:47.363865 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:47.746748 2007 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.410254 1972 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.326166 2083 raft_consensus.cc:2798] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:47.336988 1776 tablet_bootstrap.cc:654] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:47.325060 2032 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.752283 1778 tablet_bootstrap.cc:492] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:47.775177 1778 ts_tablet_manager.cc:1397] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.522s user 0.014s sys 0.042s
I20250114 02:53:47.790452 1778 raft_consensus.cc:357] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.791316 1778 raft_consensus.cc:383] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:47.791617 1778 raft_consensus.cc:738] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:47.792318 1778 consensus_queue.cc:260] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.775086 1582 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:47.783340 2083 raft_consensus.cc:695] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:47.802369 1779 raft_consensus.cc:357] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:47.806249 1702 leader_election.cc:343] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:47.805554 1779 raft_consensus.cc:383] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:47.806774 1779 raft_consensus.cc:738] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:47.807538 1779 consensus_queue.cc:260] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.774989 2007 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.776809 1972 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.819243 1778 ts_tablet_manager.cc:1428] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20250114 02:53:47.820318 1778 tablet_bootstrap.cc:492] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:47.777879 2032 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.827463 2083 consensus_queue.cc:237] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.824936 1972 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:47.812718 2090 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.828419 2032 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:47.831486 2007 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:47.839926 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 59559.5us
I20250114 02:53:47.844544 1779 ts_tablet_manager.cc:1428] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.101s user 0.008s sys 0.000s
I20250114 02:53:47.839339 2090 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.859427 1628 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:47.861088 2007 raft_consensus.cc:2743] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 02:53:47.867388 2090 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:47.861037 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:47.868387 1657 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:47.952062 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:47.954839 1776 tablet_bootstrap.cc:492] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:47.955296 1776 ts_tablet_manager.cc:1397] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.663s user 0.026s sys 0.035s
I20250114 02:53:47.967295 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:47.967989 1587 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:47.969949 2083 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:47.970666 2083 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.972013 2090 consensus_queue.cc:1035] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:47.973613 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:47.973394 1553 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:47.975003 2090 raft_consensus.cc:2798] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:47.975320 2090 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:47.975622 2090 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:47.977133 2083 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:47.979140 1778 tablet_bootstrap.cc:654] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:47.979190 1582 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:53:47.980041 1553 leader_election.cc:343] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:47.981613 1776 raft_consensus.cc:357] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.984108 1776 raft_consensus.cc:383] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:47.984369 1776 raft_consensus.cc:738] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:47.984964 1776 consensus_queue.cc:260] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:47.986702 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:48.004742 1776 ts_tablet_manager.cc:1428] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.049s user 0.005s sys 0.002s
I20250114 02:53:47.998442 1656 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:48.036506 2093 consensus_queue.cc:1035] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:48.040702 1553 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:48.042479 2104 raft_consensus.cc:2798] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:48.042852 2104 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:48.043162 2104 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:48.046929 2098 consensus_queue.cc:1035] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:48.051007 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:53:48.054390 1702 leader_election.cc:343] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:48.056088 1702 leader_election.cc:304] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:48.059777 2079 raft_consensus.cc:2743] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:48.063417 1958 consensus_queue.cc:1035] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250114 02:53:48.060739 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:48.057374 2112 consensus_queue.cc:1035] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20250114 02:53:48.067636 1654 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:48.067834 2090 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.053874 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:48.056524 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:48.086675 1702 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
W20250114 02:53:48.088703 1553 leader_election.cc:343] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:48.092674 2094 raft_consensus.cc:2798] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:48.093767 2094 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:48.094090 2094 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:48.099588 2098 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:48.099820 2090 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:48.100075 2098 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.090378 2106 consensus_queue.cc:1035] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20250114 02:53:48.103410 1776 tablet_bootstrap.cc:492] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:48.102875 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:48.092017 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:48.101295 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:48.117702 1654 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:53:48.120623 1553 leader_election.cc:343] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:48.120888 1553 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:48.121917 2106 raft_consensus.cc:2743] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:48.111091 1734 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:48.120267 2090 consensus_queue.cc:1035] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250114 02:53:48.117159 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:48.126343 1776 tablet_bootstrap.cc:654] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.140177 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:48.146155 1778 tablet_bootstrap.cc:492] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:48.146684 1778 ts_tablet_manager.cc:1397] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.327s user 0.019s sys 0.013s
I20250114 02:53:48.150871 1779 tablet_bootstrap.cc:492] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
W20250114 02:53:48.152838 1628 leader_election.cc:343] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:48.171995 1778 raft_consensus.cc:357] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:48.172853 1628 leader_election.cc:343] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:48.173110 1778 raft_consensus.cc:383] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.173458 1778 raft_consensus.cc:738] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.174111 1778 consensus_queue.cc:260] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.150942 2104 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.185197 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:48.194834 2098 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:48.197530 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:48.204067 2104 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:48.204438 1858 consensus_queue.cc:1035] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:48.206538 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:48.207141 1662 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:48.215480 1778 ts_tablet_manager.cc:1428] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.068s user 0.012s sys 0.005s
I20250114 02:53:48.216234 2099 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:48.216531 1778 tablet_bootstrap.cc:492] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:48.216792 2099 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.219166 2099 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:48.225196 1779 tablet_bootstrap.cc:654] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.206036 2094 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.232192 2094 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:48.232574 1728 raft_consensus.cc:2388] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:48.240773 1776 tablet_bootstrap.cc:492] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:48.241175 1776 ts_tablet_manager.cc:1397] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.138s user 0.008s sys 0.009s
W20250114 02:53:48.249048 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:53:48.249115 2005 consensus_queue.cc:1035] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:48.250666 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:48.253460 1553 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:48.256443 1776 raft_consensus.cc:357] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.257318 1776 raft_consensus.cc:383] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.257591 1776 raft_consensus.cc:738] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.264513 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:48.255736 1628 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
W20250114 02:53:48.267302 1628 leader_election.cc:343] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:48.267697 1628 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:48.254386 1654 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:48.270172 1972 raft_consensus.cc:2743] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:48.235285 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:48.272347 1778 tablet_bootstrap.cc:654] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.267741 1776 consensus_queue.cc:260] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:48.276834 1553 leader_election.cc:343] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:53:48.280328 1702 consensus_peers.cc:487] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 -> Peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Couldn't send request to peer 62daa58e8ff64698bb46ea3fd5290cdd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:48.273346 1958 raft_consensus.cc:2743] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:53:48.302798 1628 leader_election.cc:343] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:48.348547 1776 ts_tablet_manager.cc:1428] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.107s user 0.005s sys 0.006s
I20250114 02:53:48.352021 1776 tablet_bootstrap.cc:492] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:48.299276 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:48.357321 1553 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:48.350618 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:48.358642 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:48.349272 2005 consensus_queue.cc:1035] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20250114 02:53:48.365670 2092 raft_consensus.cc:2798] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:48.385485 1776 tablet_bootstrap.cc:654] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
W20250114 02:53:48.384334 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 86844.2us
I20250114 02:53:48.383824 1660 raft_consensus.cc:2388] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:48.397447 2092 raft_consensus.cc:695] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:48.398257 2092 consensus_queue.cc:237] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.389962 2098 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:48.400657 2098 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:48.401839 1628 leader_election.cc:343] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:48.402153 1628 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:48.404942 1702 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:48.405578 1838 raft_consensus.cc:2743] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:48.405119 2007 raft_consensus.cc:2743] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:48.427608 1779 tablet_bootstrap.cc:492] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:48.428045 1779 ts_tablet_manager.cc:1397] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.277s user 0.026s sys 0.012s
I20250114 02:53:48.430217 1662 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:48.432335 2091 raft_consensus.cc:2758] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20250114 02:53:48.432703 2127 log.cc:926] Time spent T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
W20250114 02:53:48.434197 2128 log.cc:926] Time spent T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250114 02:53:48.431301 2079 consensus_queue.cc:1035] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250114 02:53:48.438045 2098 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:48.444362 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:48.488816 1779 raft_consensus.cc:357] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.490059 1779 raft_consensus.cc:383] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.490451 1779 raft_consensus.cc:738] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
W20250114 02:53:48.521808 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 62106.9us
W20250114 02:53:48.523133 1553 leader_election.cc:343] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:48.521924 1779 consensus_queue.cc:260] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.545161 1779 ts_tablet_manager.cc:1428] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.117s user 0.004s sys 0.004s
I20250114 02:53:48.545778 1779 tablet_bootstrap.cc:492] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:48.567687 1482 catalog_manager.cc:5526] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:48.569329 1776 tablet_bootstrap.cc:492] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:48.569783 1776 ts_tablet_manager.cc:1397] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.218s user 0.017s sys 0.054s
I20250114 02:53:48.582602 1778 tablet_bootstrap.cc:492] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:48.582460 2093 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:48.583091 1778 ts_tablet_manager.cc:1397] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.367s user 0.012s sys 0.010s
I20250114 02:53:48.583065 2093 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.587342 1702 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:48.592074 1836 raft_consensus.cc:2743] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 02:53:48.594313 1484 catalog_manager.cc:5526] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:53:48.596810 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:48.597348 1661 raft_consensus.cc:2463] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:48.600077 2105 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:48.600507 2105 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.604418 1776 raft_consensus.cc:357] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.605067 1776 raft_consensus.cc:383] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.605314 1776 raft_consensus.cc:738] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.605882 1776 consensus_queue.cc:260] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.609452 2105 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:48.610580 1778 raft_consensus.cc:357] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.611295 1778 raft_consensus.cc:383] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.611560 2093 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:48.611572 1778 raft_consensus.cc:738] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.614312 1778 consensus_queue.cc:260] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:48.615156 2140 log.cc:926] Time spent T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20250114 02:53:48.615880 1778 ts_tablet_manager.cc:1428] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.032s user 0.002s sys 0.002s
I20250114 02:53:48.619158 1778 tablet_bootstrap.cc:492] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:48.623431 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:48.623656 1776 ts_tablet_manager.cc:1428] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.054s user 0.007s sys 0.002s
I20250114 02:53:48.625120 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:48.628513 1553 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:48.629603 1776 tablet_bootstrap.cc:492] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:48.630317 2138 raft_consensus.cc:2798] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
W20250114 02:53:48.630748 1628 leader_election.cc:343] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:53:48.631392 1628 leader_election.cc:343] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:48.631810 2100 consensus_queue.cc:1035] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250114 02:53:48.631850 2138 raft_consensus.cc:695] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:48.633409 2138 consensus_queue.cc:237] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.640662 1779 tablet_bootstrap.cc:654] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.640539 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:48.641783 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:48.644673 1553 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
W20250114 02:53:48.646553 1628 leader_election.cc:343] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:48.646939 2092 raft_consensus.cc:2798] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:48.649257 2092 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:48.649493 1778 tablet_bootstrap.cc:654] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.649619 2092 raft_consensus.cc:3054] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:48.663060 1778 tablet_bootstrap.cc:492] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:48.663506 1778 ts_tablet_manager.cc:1397] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.004s
I20250114 02:53:48.670497 1776 tablet_bootstrap.cc:654] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.671706 1628 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:48.672327 2093 raft_consensus.cc:2743] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:48.672217 2092 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.670491 1778 raft_consensus.cc:357] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.673925 1778 raft_consensus.cc:383] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.674199 1778 raft_consensus.cc:738] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.674413 1779 tablet_bootstrap.cc:492] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:48.674999 1583 raft_consensus.cc:2388] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:48.675136 1779 ts_tablet_manager.cc:1397] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.129s user 0.008s sys 0.003s
W20250114 02:53:48.675936 1702 consensus_peers.cc:487] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 -> Peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Couldn't send request to peer 1e5870f8269b49cf81dc39f1a5c8f5c1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:48.674801 1778 consensus_queue.cc:260] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.684268 1779 raft_consensus.cc:357] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.684845 1779 raft_consensus.cc:383] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.685086 1779 raft_consensus.cc:738] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.690673 1778 ts_tablet_manager.cc:1428] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.027s user 0.005s sys 0.002s
I20250114 02:53:48.690904 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:48.691406 1656 raft_consensus.cc:3054] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:48.691726 2092 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:48.694221 1628 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:48.698838 1779 consensus_queue.cc:260] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.700946 1779 ts_tablet_manager.cc:1428] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.025s user 0.004s sys 0.000s
I20250114 02:53:48.701642 1779 tablet_bootstrap.cc:492] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:48.703630 1656 raft_consensus.cc:2463] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:48.704093 2105 raft_consensus.cc:2743] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:48.705724 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:48.707453 1485 catalog_manager.cc:5526] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
W20250114 02:53:48.710086 1553 leader_election.cc:343] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:48.691488 1778 tablet_bootstrap.cc:492] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:48.710877 1485 catalog_manager.cc:5526] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:48.715555 1778 tablet_bootstrap.cc:654] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.722247 1779 tablet_bootstrap.cc:654] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.723057 1553 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:48.730865 2092 raft_consensus.cc:2798] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:48.731539 2092 raft_consensus.cc:695] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:48.732369 2092 consensus_queue.cc:237] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.735061 2100 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:48.735527 2100 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.780216 2100 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:48.827308 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:48.827878 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:48.828047 1727 raft_consensus.cc:2463] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:48.836055 1776 tablet_bootstrap.cc:492] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:48.836545 1776 ts_tablet_manager.cc:1397] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.212s user 0.022s sys 0.010s
I20250114 02:53:48.827373 2095 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:48.865463 1778 tablet_bootstrap.cc:492] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:48.865973 1778 ts_tablet_manager.cc:1397] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.175s user 0.017s sys 0.033s
I20250114 02:53:48.881418 1776 raft_consensus.cc:357] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.888152 1776 raft_consensus.cc:383] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.888494 1776 raft_consensus.cc:738] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.889253 1776 consensus_queue.cc:260] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.863317 2095 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.893245 1776 ts_tablet_manager.cc:1428] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.056s user 0.004s sys 0.003s
I20250114 02:53:48.894543 1776 tablet_bootstrap.cc:492] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:48.894608 2095 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:48.900807 1779 tablet_bootstrap.cc:492] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:48.901171 1779 ts_tablet_manager.cc:1397] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.200s user 0.018s sys 0.007s
I20250114 02:53:48.902163 1776 tablet_bootstrap.cc:654] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.903549 1779 raft_consensus.cc:357] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.904284 1779 raft_consensus.cc:383] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.904536 1779 raft_consensus.cc:738] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.907333 1779 consensus_queue.cc:260] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.908532 1776 tablet_bootstrap.cc:492] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:48.909053 1776 ts_tablet_manager.cc:1397] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250114 02:53:48.909893 1779 ts_tablet_manager.cc:1428] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20250114 02:53:48.910723 1779 tablet_bootstrap.cc:492] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:48.911659 1776 raft_consensus.cc:357] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.912282 1776 raft_consensus.cc:383] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.912549 1776 raft_consensus.cc:738] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.913100 1776 consensus_queue.cc:260] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.915227 1776 ts_tablet_manager.cc:1428] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250114 02:53:48.915519 1779 tablet_bootstrap.cc:654] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.915887 1776 tablet_bootstrap.cc:492] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:48.920733 1776 tablet_bootstrap.cc:654] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.920305 1779 tablet_bootstrap.cc:492] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:48.921449 1779 ts_tablet_manager.cc:1397] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20250114 02:53:48.924546 1779 raft_consensus.cc:357] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.928480 1779 raft_consensus.cc:383] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.928736 1779 raft_consensus.cc:738] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.929299 1779 consensus_queue.cc:260] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.931562 1779 ts_tablet_manager.cc:1428] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250114 02:53:48.932361 1779 tablet_bootstrap.cc:492] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:48.937321 1779 tablet_bootstrap.cc:654] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.941892 1776 tablet_bootstrap.cc:492] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:48.942283 1776 ts_tablet_manager.cc:1397] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.003s
I20250114 02:53:48.944599 1776 raft_consensus.cc:357] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.945271 1776 raft_consensus.cc:383] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.945549 1776 raft_consensus.cc:738] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.946075 1776 consensus_queue.cc:260] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.947993 1776 ts_tablet_manager.cc:1428] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250114 02:53:48.948755 1776 tablet_bootstrap.cc:492] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:48.953923 1779 tablet_bootstrap.cc:492] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:48.953977 1776 tablet_bootstrap.cc:654] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.954630 1779 ts_tablet_manager.cc:1397] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.004s
I20250114 02:53:48.957345 1779 raft_consensus.cc:357] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.957952 1779 raft_consensus.cc:383] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.958240 1779 raft_consensus.cc:738] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.958815 1779 consensus_queue.cc:260] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.960917 1779 ts_tablet_manager.cc:1428] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:53:48.961653 1779 tablet_bootstrap.cc:492] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:48.966171 1776 tablet_bootstrap.cc:492] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:48.966637 1776 ts_tablet_manager.cc:1397] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20250114 02:53:48.969524 1776 raft_consensus.cc:357] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.970216 1776 raft_consensus.cc:383] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.970474 1776 raft_consensus.cc:738] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.971276 1779 tablet_bootstrap.cc:654] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.971134 1776 consensus_queue.cc:260] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.972474 1776 ts_tablet_manager.cc:1428] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:53:48.973058 1776 tablet_bootstrap.cc:492] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:48.975360 1779 tablet_bootstrap.cc:492] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:48.975812 1779 ts_tablet_manager.cc:1397] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.001s
I20250114 02:53:48.978367 1776 tablet_bootstrap.cc:654] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.977991 1779 raft_consensus.cc:357] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.978708 1779 raft_consensus.cc:383] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.978988 1779 raft_consensus.cc:738] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.979678 1779 consensus_queue.cc:260] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.981905 1779 ts_tablet_manager.cc:1428] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250114 02:53:48.982923 1779 tablet_bootstrap.cc:492] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:48.986469 2079 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:48.986878 2079 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.988229 1776 tablet_bootstrap.cc:492] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:48.988555 2079 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:48.988612 1776 ts_tablet_manager.cc:1397] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250114 02:53:48.989979 1779 tablet_bootstrap.cc:654] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:48.990283 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:48.989599 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:48.992159 1776 raft_consensus.cc:357] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.992849 1656 raft_consensus.cc:2388] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:48.992803 1776 raft_consensus.cc:383] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:48.993232 1776 raft_consensus.cc:738] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:48.993873 1776 consensus_queue.cc:260] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:48.995810 1776 ts_tablet_manager.cc:1428] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250114 02:53:48.996557 1776 tablet_bootstrap.cc:492] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:48.996821 1779 tablet_bootstrap.cc:492] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:48.997191 1779 ts_tablet_manager.cc:1397] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20250114 02:53:48.997355 1702 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:48.998350 2095 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:49.002452 1776 tablet_bootstrap.cc:654] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:49.002676 1779 raft_consensus.cc:357] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:49.003338 1779 raft_consensus.cc:383] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:49.003603 1779 raft_consensus.cc:738] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:49.004292 1779 consensus_queue.cc:260] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:49.005263 2095 raft_consensus.cc:2743] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:49.006538 1779 ts_tablet_manager.cc:1428] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.009s user 0.002s sys 0.002s
I20250114 02:53:49.007575 1779 tablet_bootstrap.cc:492] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:49.008147 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:49.008766 1580 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:49.012766 1778 raft_consensus.cc:357] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:49.013232 1778 raft_consensus.cc:383] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:49.013490 1778 raft_consensus.cc:738] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:49.013927 1778 consensus_queue.cc:260] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:49.020413 1553 leader_election.cc:343] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:49.020963 1778 ts_tablet_manager.cc:1428] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.155s user 0.004s sys 0.000s
I20250114 02:53:49.021777 1778 tablet_bootstrap.cc:492] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:49.022739 2137 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:49.023944 2137 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:49.026175 2137 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:49.031255 1702 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:49.032649 1836 raft_consensus.cc:2798] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:49.032967 1836 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:49.033286 1836 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:49.048532 2139 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:49.049003 2139 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:49.050323 1779 tablet_bootstrap.cc:654] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:49.054716 1836 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:49.056979 1836 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:49.060050 1553 leader_election.cc:304] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:49.063725 2138 raft_consensus.cc:2798] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:49.246729 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:49.376061 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:49.522053 1659 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:49.448578 1657 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:49.067495 2092 consensus_queue.cc:1035] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.910s
I20250114 02:53:50.005383 1484 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39748 (request call id 98) took 1076 ms. Trace:
I20250114 02:53:50.006867 1485 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39724 (request call id 109) took 1275 ms. Trace:
I20250114 02:53:50.007336 1485 rpcz_store.cc:276] 0114 02:53:48.731025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:48.736201 (+ 5176us) service_pool.cc:225] Handling call
0114 02:53:50.006775 (+1270574us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1011483,"reported_tablets":2}
I20250114 02:53:50.031522 1482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39732 (request call id 89) took 1311 ms. Trace:
I20250114 02:53:50.005823 1484 rpcz_store.cc:276] 0114 02:53:48.928733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:48.928860 (+ 127us) service_pool.cc:225] Handling call
0114 02:53:50.005260 (+1076400us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28343,"reported_tablets":1}
I20250114 02:53:50.055761 1778 tablet_bootstrap.cc:654] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
W20250114 02:53:50.054742 1779 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.995s user 0.057s sys 0.093s
W20250114 02:53:50.072292 1779 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 1.013s user 0.057s sys 0.094s
I20250114 02:53:50.032277 1482 rpcz_store.cc:276] 0114 02:53:48.719162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:48.742820 (+ 23658us) service_pool.cc:225] Handling call
0114 02:53:50.030494 (+1287674us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1039986,"reported_tablets":4}
I20250114 02:53:50.046062 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 150) took 1099 ms. Trace:
I20250114 02:53:50.075765 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 203) took 1041 ms. Trace:
I20250114 02:53:50.086719 1779 tablet_bootstrap.cc:492] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:50.087255 1779 ts_tablet_manager.cc:1397] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 1.080s user 0.067s sys 0.095s
I20250114 02:53:50.091053 1779 raft_consensus.cc:357] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.094285 1779 raft_consensus.cc:383] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:50.095404 1779 raft_consensus.cc:738] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:50.097158 1779 consensus_queue.cc:260] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.082352 1728 rpcz_store.cc:276] 0114 02:53:49.034340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:49.035344 (+ 1004us) service_pool.cc:225] Handling call
0114 02:53:50.075263 (+1039919us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:50.075421 (+ 158us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:50.075447 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:50.075493 (+ 46us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:50.075523 (+ 30us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:50.075578 (+ 55us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:50.075688 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:50.106835 2179 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:50.081014 1584 rpcz_store.cc:276] 0114 02:53:48.946821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.043947 (+1097126us) service_pool.cc:225] Handling call
0114 02:53:50.044690 (+ 743us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:50.045080 (+ 390us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:50.045163 (+ 83us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:50.045296 (+ 133us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:50.045426 (+ 130us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:50.045584 (+ 158us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:50.045870 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:50.129335 2184 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:50.125656 2175 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:50.131963 2175 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.133451 2169 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.133838 2175 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.119542 2179 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.134071 2169 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.135871 2179 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.136354 2169 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.130548 2184 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.137378 2170 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.138221 2184 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.137890 2170 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.140367 1553 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:50.145505 2169 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.146836 2169 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.148617 2169 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:50.153482 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:53:50.145211 2137 raft_consensus.cc:2798] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:50.166455 1836 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.166922 1836 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.170917 2185 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.171337 2185 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.172780 2170 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.173089 2185 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.173990 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 205) took 1126 ms. Trace:
I20250114 02:53:50.165427 2137 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:50.175663 1836 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.176028 1836 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:50.176280 1836 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.145845 2187 consensus_queue.cc:1035] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20250114 02:53:50.178894 1836 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.177497 2186 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.181514 2188 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.181656 2186 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.181959 2188 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.174327 1728 rpcz_store.cc:276] 0114 02:53:50.148433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.164154 (+ 15721us) service_pool.cc:225] Handling call
0114 02:53:50.168097 (+ 3943us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:50.168829 (+ 732us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:50.170181 (+ 1352us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:50.173743 (+ 3562us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:50.173778 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:50.173835 (+ 57us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:50.173941 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:50.184437 2186 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.184895 2186 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.174911 2193 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.175525 2137 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:50.190419 2193 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.188552 2186 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.192860 1776 tablet_bootstrap.cc:492] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:50.192915 2186 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.193353 1776 ts_tablet_manager.cc:1397] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 1.197s user 0.036s sys 0.090s
I20250114 02:53:50.202171 1776 raft_consensus.cc:357] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.203128 1776 raft_consensus.cc:383] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:50.203454 1776 raft_consensus.cc:738] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:50.204069 1776 consensus_queue.cc:260] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.178309 2170 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.206154 1836 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.206269 2193 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.207203 1836 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.207646 2188 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.209626 1836 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.210191 1836 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.210669 1836 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.213172 1779 ts_tablet_manager.cc:1428] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.126s user 0.011s sys 0.007s
I20250114 02:53:50.213737 1779 tablet_bootstrap.cc:492] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
W20250114 02:53:50.214821 1683 debug-util.cc:398] Leaking SignalData structure 0x7b08000fb5a0 after lost signal to thread 428
W20250114 02:53:50.215597 1683 debug-util.cc:398] Leaking SignalData structure 0x7b08000fbe80 after lost signal to thread 1519
I20250114 02:53:50.216094 1836 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.216938 1836 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.217450 1836 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.219393 1836 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.219789 1836 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.221005 2189 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:50.220170 1836 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.221405 2189 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.225060 1776 ts_tablet_manager.cc:1428] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.031s user 0.009s sys 0.005s
I20250114 02:53:50.225683 1776 tablet_bootstrap.cc:492] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:50.212201 2170 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.228017 2170 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.229143 1836 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.229521 1836 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.229923 1836 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.232146 2196 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.228677 2190 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.228355 2137 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.235541 1778 tablet_bootstrap.cc:492] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:50.235930 1778 ts_tablet_manager.cc:1397] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 1.214s user 0.024s sys 0.016s
I20250114 02:53:50.232301 2191 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.238443 1836 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.239068 1836 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.239614 1836 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.230792 2170 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.247232 1836 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.247540 1836 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.247810 1836 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.232578 2196 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.250082 1836 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.251188 1836 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:50.251472 1836 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.251791 2198 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.252225 2198 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.252830 2200 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.252962 1836 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.253314 2200 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.253863 2198 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.254354 2198 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.254837 2194 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.255075 2196 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.256423 1836 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.255241 2194 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.254702 2198 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.246981 2170 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.257830 2189 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.241225 2190 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.264262 2137 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.258975 1836 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.264212 2200 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.270018 2188 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.271610 2188 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.266337 2198 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.272761 2198 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.266904 2189 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.276053 2193 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.277046 2197 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.273005 2194 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.277247 2193 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.279670 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.280184 2193 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.280282 1656 raft_consensus.cc:2463] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:50.281277 2199 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:50.274356 2137 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.273808 2200 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:49.174705 2138 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:50.287709 2190 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.287849 2195 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.287797 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.288102 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.276444 2189 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.289608 1778 raft_consensus.cc:357] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.292224 1778 raft_consensus.cc:383] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:50.287818 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.292526 1778 raft_consensus.cc:738] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:50.276587 2188 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.273485 2198 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.290846 2200 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.291028 1836 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.247079 2191 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.287410 2194 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.277794 2170 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.286934 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.281729 2199 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.294315 1836 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.310559 1778 consensus_queue.cc:260] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.324462 1836 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.291083 2138 raft_consensus.cc:3054] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:50.314841 2188 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.282363 2197 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.294658 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.327414 2200 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.310237 2201 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.341650 1779 tablet_bootstrap.cc:654] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:50.289376 2137 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.347501 2139 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.293411 1654 raft_consensus.cc:2463] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:50.351167 2189 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:49.178483 2103 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:50.350323 2197 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.328199 2188 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.340885 2199 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.352797 2189 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:50.335518 2194 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.347714 2201 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.374927 2191 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.374882 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.352540 2103 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.376338 2202 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.376809 2202 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.375768 2198 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.363803 2189 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.389721 1553 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:50.389881 1776 tablet_bootstrap.cc:654] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:50.397181 1836 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.294754 2195 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.402550 2005 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.404173 2189 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.385795 2202 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.404148 2005 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.386682 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.400709 2170 raft_consensus.cc:2798] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:50.384960 2194 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.413421 2170 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.413791 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.386181 2103 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.418907 2170 raft_consensus.cc:3054] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:50.405524 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.416554 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.387893 2188 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.431795 1836 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.431809 1553 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:50.434650 1836 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.432842 2201 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.433899 2195 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.442054 1778 ts_tablet_manager.cc:1428] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.206s user 0.023s sys 0.014s
I20250114 02:53:50.442901 2188 raft_consensus.cc:2798] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:50.443142 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 202) took 1425 ms. Trace:
I20250114 02:53:50.443351 2188 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.443493 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.443933 2188 raft_consensus.cc:3054] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:50.444187 1654 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:50.444766 1836 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.444083 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.446133 1778 tablet_bootstrap.cc:492] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:50.445096 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.439046 1658 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:50.447535 2137 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.447827 1654 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:50.448052 2005 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.448366 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:50.448762 1662 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:50.448971 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:50.449980 2204 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:50.456594 1836 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.433928 2170 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.457142 1836 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.471313 2180 raft_consensus.cc:491] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.443528 1727 rpcz_store.cc:276] 0114 02:53:49.034029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:49.035394 (+ 1365us) service_pool.cc:225] Handling call
0114 02:53:50.405668 (+1370274us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:50.442772 (+ 37104us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:50.442811 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:50.442853 (+ 42us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:50.442890 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:50.442955 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:50.443070 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:50.473050 2189 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.486589 2205 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.486653 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:50.473809 2005 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.487298 1653 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:50.483904 2200 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.490221 2194 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.471992 2180 raft_consensus.cc:513] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.493145 2194 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.492668 2005 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.486653 2191 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.488307 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:50.471933 1553 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:50.501758 1553 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:50.503264 1836 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.450479 2204 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.502840 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:50.505970 1553 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:50.506819 1836 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:50.507251 1836 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.509094 2136 raft_consensus.cc:2798] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:50.509011 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:50.509579 2136 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.510030 2136 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:50.499537 2201 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.489641 2189 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.513861 2202 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.487183 2205 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.506212 1584 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:50.495170 2200 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.505512 2005 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.585773 2202 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.633996 2100 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.513159 2103 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:50.512992 2195 raft_consensus.cc:2798] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:50.507051 2137 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.668618 1836 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:50.669211 1836 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:50.586827 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:50.677784 1836 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.503861 2191 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.630231 2193 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.641865 2100 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.651443 2186 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:50.657178 2005 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:50.653071 2170 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.655404 2195 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:50.652613 2103 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:50.587257 2201 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.682027 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:50.654238 2196 consensus_queue.cc:1035] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20250114 02:53:50.700929 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:50.706933 2195 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:50.715977 2103 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:50.671804 2137 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.714891 2180 leader_election.cc:290] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.732738 2170 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:50.716194 2005 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:50.812824 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:50.701153 2193 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.040730 1578 raft_consensus.cc:2463] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:51.041229 2194 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.041605 2202 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:50.715243 2186 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.039417 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:51.090862 2194 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:51.091770 2196 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:51.092931 2191 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:53:51.096387 2188 consensus_meta.cc:287] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.652s user 0.013s sys 0.004s
W20250114 02:53:51.097075 2138 consensus_meta.cc:287] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.762s user 0.014s sys 0.008s
I20250114 02:53:50.791831 1728 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:51.098291 2201 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:53:51.121856 1662 consensus_meta.cc:287] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.673s user 0.002s sys 0.006s
I20250114 02:53:51.118877 1836 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.160571 1836 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:51.138911 2137 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.140498 2186 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.132005 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:51.181998 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:51.182989 1580 raft_consensus.cc:2463] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:51.184039 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:51.185135 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:51.158601 2201 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:51.146668 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.147449 2138 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.149417 2005 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.129153 2194 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.004841 2170 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.151151 2100 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.132429 2196 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.158011 2191 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:51.142609 2188 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.158464 2193 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.158979 1662 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:51.186671 2137 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:51.222710 2137 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.224815 2137 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.187069 2201 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.228278 2201 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.195058 1728 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:51.230213 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.231504 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.232198 1728 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:51.235087 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.236366 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.144402 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:51.196625 2138 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.198633 2194 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.200112 2170 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.200811 2191 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.202232 2196 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.213330 2188 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.186533 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:51.079159 1654 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:51.247630 1779 tablet_bootstrap.cc:492] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:51.296895 1779 ts_tablet_manager.cc:1397] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 1.083s user 0.034s sys 0.103s
I20250114 02:53:51.295809 1728 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
W20250114 02:53:51.254843 1584 consensus_meta.cc:287] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.600s user 0.007s sys 0.001s
W20250114 02:53:51.256458 1653 consensus_meta.cc:287] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.764s user 0.009s sys 0.000s
I20250114 02:53:51.115993 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.260326 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:53:51.261109 2136 consensus_meta.cc:287] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.751s user 0.006s sys 0.003s
I20250114 02:53:51.164734 1836 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.172045 1776 tablet_bootstrap.cc:492] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:51.300176 1776 ts_tablet_manager.cc:1397] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 1.075s user 0.032s sys 0.039s
W20250114 02:53:51.261945 1701 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.005s sys 0.009s
I20250114 02:53:51.284587 1778 tablet_bootstrap.cc:654] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
W20250114 02:53:51.284828 1553 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.000s sys 0.011s
W20250114 02:53:51.288836 1551 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.003s sys 0.015s
W20250114 02:53:51.284519 1627 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.007s sys 0.004s
I20250114 02:53:51.261940 2195 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:51.303118 1553 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.651s user 0.000s sys 0.011s
W20250114 02:53:51.308009 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:53:51.307337 1656 raft_consensus.cc:2463] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
W20250114 02:53:51.306641 1627 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.572s user 0.007s sys 0.004s
I20250114 02:53:51.311530 1836 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.300756 1584 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:51.351444 2191 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.296322 1578 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:51.304718 2136 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.300627 1653 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
W20250114 02:53:51.314275 1551 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.646s user 0.003s sys 0.015s
W20250114 02:53:51.308434 1701 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.672s user 0.006s sys 0.010s
I20250114 02:53:51.367424 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:51.400345 1727 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:51.363070 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.362573 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.304113 1580 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:51.298468 2103 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:51.403236 1552 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.015s sys 0.007s
I20250114 02:53:51.407891 1728 raft_consensus.cc:2463] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
W20250114 02:53:51.410413 1552 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.667s user 0.016s sys 0.007s
I20250114 02:53:51.411307 1779 raft_consensus.cc:357] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.411970 1779 raft_consensus.cc:383] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:51.412200 1779 raft_consensus.cc:738] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:51.412691 1779 consensus_queue.cc:260] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.413982 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:51.414283 1776 raft_consensus.cc:357] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.414975 1776 raft_consensus.cc:383] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:51.415051 2103 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.415242 1776 raft_consensus.cc:738] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:51.415795 1776 consensus_queue.cc:260] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.415462 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:51.415905 1585 raft_consensus.cc:2388] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:51.418753 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.421618 1653 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:51.421947 2195 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.422713 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.419924 2191 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:51.425567 1727 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:51.426141 2136 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.426332 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.412135 1662 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:51.425714 2191 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.423466 1653 raft_consensus.cc:2463] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:51.410415 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.429000 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.429340 2191 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.429605 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.430792 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.430900 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.431310 1727 raft_consensus.cc:2463] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:51.430094 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.502997 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.504619 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.497160 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:51.505959 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.505983 1578 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:51.414011 2209 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:51.515775 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.516948 1653 raft_consensus.cc:2463] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:51.514374 2218 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:51.518462 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.519184 2218 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.520660 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.522650 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.522859 2218 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.524688 2209 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.433416 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.530004 1776 ts_tablet_manager.cc:1428] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.229s user 0.014s sys 0.008s
I20250114 02:53:51.530998 1778 tablet_bootstrap.cc:492] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:51.531908 1778 ts_tablet_manager.cc:1397] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 1.086s user 0.023s sys 0.026s
I20250114 02:53:51.531731 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.532326 1776 tablet_bootstrap.cc:492] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:51.532568 1727 raft_consensus.cc:2463] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:51.430137 1779 ts_tablet_manager.cc:1428] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.133s user 0.002s sys 0.004s
I20250114 02:53:51.431569 1728 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:51.440503 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.536593 1735 raft_consensus.cc:2463] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
W20250114 02:53:51.537123 1553 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.106.67:42283 timed out after 2.156s (SENT) [suppressed 4 similar messages]
I20250114 02:53:51.433524 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.538100 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:51.543344 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.544337 1779 tablet_bootstrap.cc:492] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:51.555675 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.557585 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:51.562005 1735 raft_consensus.cc:3054] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:53:51.563047 1628 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 2.386s user 0.021s sys 0.001s
I20250114 02:53:51.564589 1778 raft_consensus.cc:357] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.552383 2209 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.567896 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.571157 1778 raft_consensus.cc:383] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:51.571887 1778 raft_consensus.cc:738] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:51.572770 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.569350 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:51.568280 1628 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 2.391s user 0.021s sys 0.002s
I20250114 02:53:51.575024 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:53:51.576823 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 2.39994e+06us
I20250114 02:53:51.577155 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.577927 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.578851 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:51.572798 1778 consensus_queue.cc:260] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.584743 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 154) took 1312 ms. Trace:
I20250114 02:53:51.585520 1657 rpcz_store.cc:276] 0114 02:53:50.275434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.276540 (+ 1106us) service_pool.cc:225] Handling call
0114 02:53:50.277194 (+ 654us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:50.293314 (+ 16120us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:50.293803 (+ 489us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:50.294393 (+ 590us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:50.294435 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:51.581623 (+1287188us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b54000c1a54. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:53:51.581688 (+ 65us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:51.581976 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36729216}
I20250114 02:53:51.587486 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 170) took 1200 ms. Trace:
I20250114 02:53:51.587771 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.592741 1657 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:51.593145 1778 ts_tablet_manager.cc:1428] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.061s user 0.009s sys 0.002s
I20250114 02:53:51.593695 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.593878 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.594512 1729 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:51.595353 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.595559 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.596352 1657 raft_consensus.cc:2388] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:51.596251 1729 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:51.596504 1779 tablet_bootstrap.cc:654] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:51.587536 1662 raft_consensus.cc:3054] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:51.573122 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:51.597170 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.601156 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 208) took 1451 ms. Trace:
I20250114 02:53:51.597421 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.598098 1658 rpcz_store.cc:276] 0114 02:53:50.387006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.448000 (+ 60994us) service_pool.cc:225] Handling call
0114 02:53:51.583976 (+1135976us) spinlock_profiling.cc:243] Waited 1.62 ms on lock 0x7b5800002d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:53:51.585552 (+ 1576us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:51.586961 (+ 1409us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:51.586991 (+ 30us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:51.587048 (+ 57us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:51.587100 (+ 52us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:51.587168 (+ 68us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:51.587268 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3717504}
I20250114 02:53:51.604979 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.605598 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.605762 1653 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:51.605875 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.606774 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:51.606971 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.607195 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:51.607501 1653 raft_consensus.cc:3054] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:51.608281 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.608434 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.611132 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.613924 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.615126 1836 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:51.601722 1732 rpcz_store.cc:276] 0114 02:53:50.149416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:51.310577 (+1161161us) service_pool.cc:225] Handling call
0114 02:53:51.598937 (+288360us) spinlock_profiling.cc:243] Waited 47.9 ms on lock 0x7b5800023d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:53:51.599342 (+ 405us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:51.599759 (+ 417us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:51.599798 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:51.599846 (+ 48us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:51.599894 (+ 48us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:51.599972 (+ 78us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:51.600954 (+ 982us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110171520}
I20250114 02:53:51.603292 1727 raft_consensus.cc:2463] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:51.615880 1836 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.616978 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.618081 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.618194 1836 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.619596 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.620419 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.621315 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.606992 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.629777 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.630215 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:51.630841 1734 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:51.631209 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:51.623132 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.597179 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.639130 1776 tablet_bootstrap.cc:654] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:51.639089 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.640942 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.641844 1662 raft_consensus.cc:2463] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:51.642144 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.677191 1836 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:51.679473 2169 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:51.680090 2169 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.681363 1734 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:51.682771 1836 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.683097 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.684011 1657 raft_consensus.cc:2388] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:51.684677 1836 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.684906 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:51.685155 1836 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:51.685689 1836 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.686245 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.687474 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.688310 1657 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:51.675696 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.676656 1735 raft_consensus.cc:2463] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:51.676959 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.690690 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.694173 2169 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.692795 1836 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625)
I20250114 02:53:51.697425 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 158) took 1417 ms. Trace:
I20250114 02:53:51.698344 1653 raft_consensus.cc:2463] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:51.691915 2189 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.679070 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.704480 1702 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:51.705837 1836 raft_consensus.cc:2798] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:51.706187 1836 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:51.705943 1702 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:51.706488 1836 raft_consensus.cc:3054] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:51.707181 2214 raft_consensus.cc:2798] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:51.707790 2214 raft_consensus.cc:695] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:51.709084 2214 consensus_queue.cc:237] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.712733 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.712116 1653 raft_consensus.cc:1235] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:51.718132 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 176) took 1131 ms. Trace:
I20250114 02:53:51.718730 1702 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:51.705152 1658 raft_consensus.cc:2463] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:51.718645 1585 rpcz_store.cc:276] 0114 02:53:50.585711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.784177 (+198466us) service_pool.cc:225] Handling call
0114 02:53:51.714257 (+930080us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b54000ca8d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:51.717120 (+ 2863us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":559071104}
I20250114 02:53:51.720046 1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 146) took 2474 ms. Trace:
I20250114 02:53:51.716470 2169 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:51.723829 2189 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:51.724234 2189 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:51.724007 2169 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.724586 2189 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:51.726815 1836 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.721603 1661 rpcz_store.cc:276] 0114 02:53:49.244783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:49.374134 (+129351us) service_pool.cc:225] Handling call
0114 02:53:51.718282 (+2344148us) spinlock_profiling.cc:243] Waited 6.43 ms on lock 0x7b5800002d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:53:51.718594 (+ 312us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:51.718915 (+ 321us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:51.718957 (+ 42us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:51.719023 (+ 66us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:51.719054 (+ 31us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:51.719105 (+ 51us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:51.719201 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14785280}
I20250114 02:53:51.729104 1836 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.697861 1655 rpcz_store.cc:276] 0114 02:53:50.279928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.280994 (+ 1066us) service_pool.cc:225] Handling call
0114 02:53:50.281258 (+ 264us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:53:51.577554 (+1296296us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7b180021ae40. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce5d6008c 00007fbce317ebfc 00007fbcebd907f9 00007fbce319ba6b 00007fbce3199e6f 00007fbce3199305 00007fbce60cd3d0 00007fbce60cdf6b 00007fbce60d878b
0114 02:53:51.577875 (+ 321us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:51.577911 (+ 36us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:53:51.583619 (+ 5708us) log.cc:844] Serialized 26 byte log entry
0114 02:53:51.646206 (+ 62587us) raft_consensus.cc:1640] Marking committed up to 0
0114 02:53:51.646544 (+ 338us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:51.648007 (+ 1463us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:53:51.697009 (+ 49002us) raft_consensus.cc:1680] finished
0114 02:53:51.697034 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:51.697275 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":46248,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":6100,"spinlock_wait_cycles":3337984,"thread_start_us":114038,"threads_started":2,"wal-append.queue_time_us":106712}
I20250114 02:53:51.731350 2169 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.731465 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.733248 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.736152 1702 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:51.736912 2083 raft_consensus.cc:2798] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:51.737162 2083 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:51.737376 2083 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:51.738104 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.743398 1702 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:51.747140 2083 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.748894 2205 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.749019 2083 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.749150 2099 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:51.749913 2099 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.750782 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:51.751370 1658 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:51.751943 2099 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.752048 2204 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.754122 2099 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:51.754962 2099 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.756656 2204 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:51.763226 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.761871 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.764142 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.761267 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.765648 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:51.760393 2204 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:51.767534 2204 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:51.767824 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.769833 1779 tablet_bootstrap.cc:492] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:51.770275 1779 ts_tablet_manager.cc:1397] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.226s user 0.016s sys 0.017s
I20250114 02:53:51.775244 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 147) took 1188 ms. Trace:
I20250114 02:53:51.759341 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:51.766321 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:51.774451 1779 raft_consensus.cc:357] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.780637 1779 raft_consensus.cc:383] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:51.780831 1779 raft_consensus.cc:738] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:51.775696 1731 rpcz_store.cc:276] 0114 02:53:50.586515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.632758 (+ 46243us) service_pool.cc:225] Handling call
0114 02:53:51.774029 (+1141271us) spinlock_profiling.cc:243] Waited 7.67 ms on lock 0x7b5800023d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:53:51.774341 (+ 312us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:51.774670 (+ 329us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:51.774707 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:51.774765 (+ 58us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:51.774819 (+ 54us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:51.774878 (+ 59us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:51.775029 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17630464}
I20250114 02:53:51.782997 2099 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:51.782284 1779 consensus_queue.cc:260] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.793354 1779 ts_tablet_manager.cc:1428] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20250114 02:53:51.794663 1779 tablet_bootstrap.cc:492] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
W20250114 02:53:51.780023 1659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48632 (request call id 155) took 1501 ms (client timeout 1918 ms). Trace:
I20250114 02:53:51.800127 2204 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.802878 2204 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.804970 1658 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:51.801445 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.805747 2189 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:51.800601 1659 rpcz_store.cc:269] 0114 02:53:50.278903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:51.765834 (+1486931us) service_pool.cc:225] Handling call
0114 02:53:51.779955 (+ 14121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:51.808029 2032 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:51.808609 2032 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:51.812556 1779 tablet_bootstrap.cc:654] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:51.808993 2189 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.813252 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:51.905939 2200 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:51.961082 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 159) took 1498 ms. Trace:
I20250114 02:53:51.961467 1587 rpcz_store.cc:276] 0114 02:53:50.462315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.467142 (+ 4827us) service_pool.cc:225] Handling call
0114 02:53:51.960460 (+1493318us) spinlock_profiling.cc:243] Waited 3.05 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:53:51.960917 (+ 457us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7011072}
I20250114 02:53:51.991868 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 206) took 1843 ms. Trace:
I20250114 02:53:51.992153 1730 rpcz_store.cc:276] 0114 02:53:50.148762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.155222 (+ 6460us) service_pool.cc:225] Handling call
0114 02:53:51.991173 (+1835951us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b5800023d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:53:51.991383 (+ 210us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:51.991560 (+ 177us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:51.991574 (+ 14us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:51.991607 (+ 33us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:51.991632 (+ 25us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:51.991668 (+ 36us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:51.991755 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5577984}
I20250114 02:53:52.505635 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 166) took 2001 ms. Trace:
I20250114 02:53:52.506165 1582 rpcz_store.cc:276] 0114 02:53:50.503876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.505592 (+ 1716us) service_pool.cc:225] Handling call
0114 02:53:52.504484 (+1998892us) spinlock_profiling.cc:243] Waited 69.2 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:53:52.504772 (+ 288us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:52.505064 (+ 292us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:52.505115 (+ 51us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:52.505163 (+ 48us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:52.505216 (+ 53us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:52.505300 (+ 84us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:52.505474 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":159256832}
I20250114 02:53:52.513257 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:52.514180 1579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 168) took 2009 ms (client timeout 2138 ms). Trace:
I20250114 02:53:52.529266 1581 raft_consensus.cc:1235] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:52.529973 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 167) took 2025 ms. Trace:
I20250114 02:53:52.530272 1581 rpcz_store.cc:276] 0114 02:53:50.504114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.505674 (+ 1560us) service_pool.cc:225] Handling call
0114 02:53:52.528782 (+2023108us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:53:52.529028 (+ 246us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:52.529710 (+ 682us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:52.529911 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26851200}
W20250114 02:53:52.514525 1579 rpcz_store.cc:269] 0114 02:53:50.504309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.511275 (+ 6966us) service_pool.cc:225] Handling call
0114 02:53:52.513038 (+2001763us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315ed0e 0000000000353db9 00007fbceafbea50 00007fbceb084814 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:53:52.514102 (+ 1064us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":45951360}
W20250114 02:53:52.562587 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 754800us
W20250114 02:53:52.563477 1628 leader_election.cc:343] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:52.563426 2191 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:52.567140 2191 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.572010 2233 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:52.569340 2191 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:52.577986 1779 tablet_bootstrap.cc:492] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
W20250114 02:53:52.578253 1628 leader_election.cc:343] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:52.578485 1779 ts_tablet_manager.cc:1397] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.784s user 0.018s sys 0.006s
I20250114 02:53:52.581430 1779 raft_consensus.cc:357] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.582123 1779 raft_consensus.cc:383] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:52.582485 1779 raft_consensus.cc:738] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:52.583205 1779 consensus_queue.cc:260] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.581719 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 161) took 2118 ms. Trace:
I20250114 02:53:52.584081 1778 tablet_bootstrap.cc:492] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:52.584265 1583 rpcz_store.cc:276] 0114 02:53:50.463123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.467074 (+ 3951us) service_pool.cc:225] Handling call
0114 02:53:52.580904 (+2113830us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315ed0e 0000000000353db9 00007fbceafbea50 00007fbceb084814 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:53:52.581099 (+ 195us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:52.581314 (+ 215us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:52.581338 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:52.581387 (+ 49us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:52.581417 (+ 30us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:52.581469 (+ 52us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:52.581562 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3385088}
I20250114 02:53:52.572624 2233 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.585911 1779 ts_tablet_manager.cc:1428] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 02:53:52.586894 1779 tablet_bootstrap.cc:492] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:52.589663 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.586891 2233 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:52.590656 1778 tablet_bootstrap.cc:654] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:52.591112 1581 raft_consensus.cc:1235] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:52.592187 1779 tablet_bootstrap.cc:654] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:52.592075 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.590374 1579 raft_consensus.cc:2463] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:52.595548 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.595999 1778 tablet_bootstrap.cc:492] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:52.596355 1778 ts_tablet_manager.cc:1397] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20250114 02:53:52.598695 1778 raft_consensus.cc:357] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.599397 1778 raft_consensus.cc:383] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:52.599660 1778 raft_consensus.cc:738] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:52.600279 1778 consensus_queue.cc:260] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.601755 1778 ts_tablet_manager.cc:1428] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250114 02:53:52.597534 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.602620 1778 tablet_bootstrap.cc:492] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:52.602269 1579 raft_consensus.cc:2463] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:52.605752 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.606524 1585 raft_consensus.cc:2388] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:52.608261 1778 tablet_bootstrap.cc:654] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:52.608407 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.609966 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:52.610968 1733 raft_consensus.cc:2388] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 1: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:52.610870 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.611552 1579 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:52.612089 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:52.612865 1730 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:52.616015 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:52.621192 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:52.621963 1778 tablet_bootstrap.cc:492] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:52.622398 1778 ts_tablet_manager.cc:1397] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.000s
I20250114 02:53:52.624845 1778 raft_consensus.cc:357] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.625449 1778 raft_consensus.cc:383] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:52.625692 1778 raft_consensus.cc:738] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:52.626726 1778 consensus_queue.cc:260] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.628463 1778 ts_tablet_manager.cc:1428] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:53:52.629417 1778 tablet_bootstrap.cc:492] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:52.631193 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.631726 1583 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:52.632580 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.634153 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.635517 1582 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:52.634897 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.633702 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.637284 1585 raft_consensus.cc:2388] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:52.634456 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.637967 1579 raft_consensus.cc:2463] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:52.638823 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.640054 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.640161 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:52.643090 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.643678 1587 raft_consensus.cc:2463] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:52.644659 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.645104 1587 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:52.645622 1628 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:52.647040 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:52.648602 2114 raft_consensus.cc:2798] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.649016 2114 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:52.649463 2114 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:53:52.638918 1660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48636 (request call id 159) took 2127 ms (client timeout 2227 ms). Trace:
W20250114 02:53:52.652436 1660 rpcz_store.cc:269] 0114 02:53:50.631575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.642622 (+ 11047us) service_pool.cc:225] Handling call
0114 02:53:51.767537 (+1124915us) spinlock_profiling.cc:243] Waited 9.53 ms on lock 0x7b5800002d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:53:52.638540 (+871003us) spinlock_profiling.cc:243] Waited 3 ms on lock 0x7b540008a628. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe184b4 00007fbceb034427 00007fbceb033bb8 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236
0114 02:53:52.638840 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28800640}
I20250114 02:53:52.653110 2092 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:52.653693 2092 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.655118 1778 tablet_bootstrap.cc:654] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:52.655539 2092 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:52.657846 1628 leader_election.cc:343] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:52.658520 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.660367 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.661934 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.663726 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.664469 1776 tablet_bootstrap.cc:492] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:52.664908 1776 ts_tablet_manager.cc:1397] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 1.133s user 0.029s sys 0.096s
I20250114 02:53:52.664830 2114 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.665396 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.666114 1587 raft_consensus.cc:2388] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:52.666810 1778 tablet_bootstrap.cc:492] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:52.667194 1778 ts_tablet_manager.cc:1397] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.007s
I20250114 02:53:52.667068 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.668367 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.669711 1778 raft_consensus.cc:357] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.670034 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.670423 1778 raft_consensus.cc:383] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:52.670763 1587 raft_consensus.cc:2388] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:52.670727 1778 raft_consensus.cc:738] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:52.671319 1778 consensus_queue.cc:260] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:52.672613 1683 debug-util.cc:398] Leaking SignalData structure 0x7b08002b8520 after lost signal to thread 1611
I20250114 02:53:52.672807 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.673381 1778 ts_tablet_manager.cc:1428] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 02:53:52.673556 1583 raft_consensus.cc:2363] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2.
W20250114 02:53:52.673727 1683 debug-util.cc:398] Leaking SignalData structure 0x7b08002b8d60 after lost signal to thread 1686
I20250114 02:53:52.674271 1778 tablet_bootstrap.cc:492] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:52.674612 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.675170 1583 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:52.675112 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:52.675810 1729 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.676379 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.679021 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.678347 1776 raft_consensus.cc:357] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.680024 1776 raft_consensus.cc:383] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:52.680342 1776 raft_consensus.cc:738] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:52.681962 1776 consensus_queue.cc:260] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:52.684662 1683 debug-util.cc:398] Leaking SignalData structure 0x7b0800281680 after lost signal to thread 1760
W20250114 02:53:52.686735 1683 debug-util.cc:398] Leaking SignalData structure 0x7b080023d560 after lost signal to thread 2118
W20250114 02:53:52.687031 1683 debug-util.cc:398] Leaking SignalData structure 0x7b080023d4e0 after lost signal to thread 2121
I20250114 02:53:52.687179 1779 tablet_bootstrap.cc:492] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:52.687618 1779 ts_tablet_manager.cc:1397] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.101s user 0.025s sys 0.026s
I20250114 02:53:52.678176 2114 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:52.693854 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 911369us
I20250114 02:53:52.698293 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.691856 1779 raft_consensus.cc:357] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.698647 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.699100 1779 raft_consensus.cc:383] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:52.699354 1586 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:52.699433 1779 raft_consensus.cc:738] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:52.699668 1702 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
W20250114 02:53:52.699892 1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 171) took 2186 ms (client timeout 2210 ms). Trace:
I20250114 02:53:52.700497 1776 ts_tablet_manager.cc:1428] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.035s user 0.003s sys 0.003s
I20250114 02:53:52.700167 1779 consensus_queue.cc:260] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.700693 2191 raft_consensus.cc:2798] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:52.701033 2191 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:52.701321 2191 raft_consensus.cc:3054] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:52.699524 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.702643 1779 ts_tablet_manager.cc:1428] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20250114 02:53:52.696611 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.701397 1776 tablet_bootstrap.cc:492] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:52.696421 2032 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:52.698102 1729 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:52.707991 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.709517 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.710978 1702 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:52.712183 2189 raft_consensus.cc:2798] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.712522 2189 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:52.712687 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:52.713449 1733 raft_consensus.cc:2463] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:52.712925 2189 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:53:52.687300 1683 debug-util.cc:398] Leaking SignalData structure 0x7b080023d5e0 after lost signal to thread 2124
I20250114 02:53:52.712531 1702 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
W20250114 02:53:52.700256 1586 rpcz_store.cc:269] 0114 02:53:50.513044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:50.513169 (+ 125us) service_pool.cc:225] Handling call
0114 02:53:52.698353 (+2185184us) spinlock_profiling.cc:243] Waited 47 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:53:52.699827 (+ 1474us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":108082944}
I20250114 02:53:52.714388 2176 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:52.718283 2176 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.720151 2176 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:52.721705 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.721819 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.722438 1583 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
W20250114 02:53:52.723068 1683 debug-util.cc:398] Leaking SignalData structure 0x7b080023d340 after lost signal to thread 2134
W20250114 02:53:52.723793 1683 debug-util.cc:398] Leaking SignalData structure 0x7b080023d400 after lost signal to thread 2149
I20250114 02:53:52.723904 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.725149 2095 raft_consensus.cc:2798] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:52.725548 2095 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:52.725283 2189 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.725863 2095 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:52.727097 2189 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:52.730947 1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 187) took 1127 ms. Trace:
I20250114 02:53:52.731284 1661 rpcz_store.cc:276] 0114 02:53:51.624885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:51.732360 (+107475us) service_pool.cc:225] Handling call
0114 02:53:52.730250 (+997890us) spinlock_profiling.cc:243] Waited 5.47 ms on lock 0x7b5800002d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:53:52.730439 (+ 189us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:52.730643 (+ 204us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:52.730667 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:52.730715 (+ 48us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:52.730746 (+ 31us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:52.730795 (+ 49us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:52.730894 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12580608}
I20250114 02:53:52.732307 2095 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.735013 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:52.737761 1628 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:52.739979 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.740192 2182 raft_consensus.cc:2798] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:52.740674 2182 raft_consensus.cc:695] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:52.741423 2182 consensus_queue.cc:237] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:52.742244 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 1.20533e+06us
I20250114 02:53:52.743854 1553 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:52.742920 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.744978 2092 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.745296 2092 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:52.745642 2092 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.736811 1734 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:52.746891 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.747490 1583 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:52.750830 1778 tablet_bootstrap.cc:654] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:52.751510 1779 tablet_bootstrap.cc:492] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:52.753691 1702 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:52.754400 1628 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:52.755779 2232 raft_consensus.cc:2798] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.756161 2232 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:52.756538 2232 raft_consensus.cc:3054] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.759877 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.760680 1587 raft_consensus.cc:2388] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:52.762019 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:52.762540 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.763212 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.763800 1581 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:52.765806 2232 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:52.766839 1628 leader_election.cc:343] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:52.767359 1628 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.769331 2204 raft_consensus.cc:2758] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250114 02:53:52.771924 1553 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:52.772782 2106 raft_consensus.cc:2798] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.773142 2106 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:52.773510 2106 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.773622 2189 consensus_queue.cc:1046] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250114 02:53:52.775936 2204 raft_consensus.cc:3049] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:52.776307 2204 raft_consensus.cc:738] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
W20250114 02:53:52.776435 1628 leader_election.cc:343] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:52.752300 2191 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.776903 1628 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.777222 2173 raft_consensus.cc:2743] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:52.777655 2197 raft_consensus.cc:2743] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:52.777990 1628 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.779762 2197 raft_consensus.cc:2743] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.779364 2191 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:52.780213 1628 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.780916 2197 raft_consensus.cc:2743] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.781280 1628 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.781968 2197 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.782176 2095 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:52.782361 1628 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.775372 1702 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:52.778203 2204 consensus_queue.cc:260] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.788681 2182 raft_consensus.cc:2743] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.790360 2204 raft_consensus.cc:3054] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.794030 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.794247 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:52.794911 2197 raft_consensus.cc:2743] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.794993 2092 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.795935 1628 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.796797 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:52.796919 2092 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:52.796998 1628 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.799021 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:52.806005 2182 raft_consensus.cc:2743] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.806203 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:52.806852 1660 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.806910 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:52.806972 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.800359 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:52.811115 1776 tablet_bootstrap.cc:654] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:52.813828 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:52.806300 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:52.814949 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:52.815470 1653 raft_consensus.cc:3054] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:52.816066 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:52.817107 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:52.814997 1583 raft_consensus.cc:1235] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:52.819425 2232 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:52.819921 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:52.820233 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:52.820587 1659 raft_consensus.cc:2463] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:52.821511 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:52.823182 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:52.808430 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:52.823776 1655 raft_consensus.cc:2463] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 0.
I20250114 02:53:52.787623 2191 raft_consensus.cc:2798] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.829157 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:52.832557 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:52.807837 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:52.827134 2197 raft_consensus.cc:2743] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.839583 1628 leader_election.cc:400] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2.
I20250114 02:53:52.822311 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:52.783389 2106 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:52.847290 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 59913.9us
I20250114 02:53:52.846385 1628 leader_election.cc:403] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
I20250114 02:53:52.848608 1702 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:52.846266 1580 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:52.854751 2182 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.856925 1628 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:52.858371 2106 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:52.860416 1778 tablet_bootstrap.cc:492] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:52.860875 1778 ts_tablet_manager.cc:1397] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.187s user 0.004s sys 0.023s
I20250114 02:53:52.861019 2189 raft_consensus.cc:2743] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.862965 1778 raft_consensus.cc:357] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.863471 1778 raft_consensus.cc:383] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:52.863708 1778 raft_consensus.cc:738] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:52.864184 1778 consensus_queue.cc:260] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.865332 2182 raft_consensus.cc:2743] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2.
I20250114 02:53:52.865793 1778 ts_tablet_manager.cc:1428] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250114 02:53:52.846536 1662 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:52.866480 1778 tablet_bootstrap.cc:492] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:52.846800 1654 raft_consensus.cc:3054] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:52.844218 2191 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:52.845937 1653 raft_consensus.cc:2463] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:52.871337 1628 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.873299 2197 raft_consensus.cc:2798] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.873699 2197 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:52.869943 2191 raft_consensus.cc:3054] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.874099 1553 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:52.873992 1660 raft_consensus.cc:2463] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:53:52.875897 1628 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.874789 2197 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.878268 1779 tablet_bootstrap.cc:654] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:52.878690 2092 raft_consensus.cc:2798] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.879060 2092 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:52.879426 2092 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.879808 1553 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:52.880090 1580 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:52.880090 1654 raft_consensus.cc:2463] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
W20250114 02:53:52.878288 1702 leader_election.cc:343] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:52.882262 1702 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:52.882936 2232 raft_consensus.cc:2743] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.884231 1628 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.887125 1776 tablet_bootstrap.cc:492] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:52.887410 2232 raft_consensus.cc:2743] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.887593 1776 ts_tablet_manager.cc:1397] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.186s user 0.019s sys 0.019s
I20250114 02:53:52.884117 2197 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.886099 2106 raft_consensus.cc:2798] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.888926 2106 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:52.889325 2106 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.890863 1776 raft_consensus.cc:357] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.891499 1776 raft_consensus.cc:383] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:52.891773 1776 raft_consensus.cc:738] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:52.892390 1776 consensus_queue.cc:260] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.893049 1778 tablet_bootstrap.cc:654] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:52.886427 1553 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:52.888381 1628 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.895205 2197 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:52.895440 2106 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.897079 2106 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:52.898501 1553 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.899492 1702 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:52.875465 2182 raft_consensus.cc:2743] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.903591 2244 raft_consensus.cc:2798] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.890369 1662 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:52.904073 2197 raft_consensus.cc:2743] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.903626 2242 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:52.904223 1628 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.904668 2242 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.905283 2197 raft_consensus.cc:2743] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.905478 1628 leader_election.cc:304] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.903307 2106 raft_consensus.cc:2798] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:52.910840 2232 raft_consensus.cc:2743] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:53:52.911428 1628 leader_election.cc:343] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:52.912499 1628 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.912973 2242 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:52.913167 2232 raft_consensus.cc:2743] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:52.904148 2244 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:52.913691 2244 raft_consensus.cc:3054] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.891944 2092 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.914841 1628 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:52.915311 1702 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:52.918596 2242 raft_consensus.cc:2743] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.919035 1628 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
W20250114 02:53:52.919941 1516 debug-util.cc:398] Leaking SignalData structure 0x7b08000399a0 after lost signal to thread 428
W20250114 02:53:52.921087 1516 debug-util.cc:398] Leaking SignalData structure 0x7b080003e080 after lost signal to thread 1519
I20250114 02:53:52.924038 1628 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.923110 2232 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250114 02:53:52.925343 1516 debug-util.cc:398] Leaking SignalData structure 0x7b0800282500 after lost signal to thread 1611
I20250114 02:53:52.925614 2232 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:52.926107 2232 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:53:52.926513 1516 debug-util.cc:398] Leaking SignalData structure 0x7b0800233560 after lost signal to thread 1686
W20250114 02:53:52.926858 1516 debug-util.cc:398] Leaking SignalData structure 0x7b0800233580 after lost signal to thread 1687
W20250114 02:53:52.927819 1516 debug-util.cc:398] Leaking SignalData structure 0x7b080023c900 after lost signal to thread 1760
I20250114 02:53:52.911432 2189 raft_consensus.cc:2743] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.932438 2095 raft_consensus.cc:2798] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.932842 2095 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:52.933215 2095 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.922787 2182 raft_consensus.cc:2743] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.938531 1628 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.936645 2197 raft_consensus.cc:2743] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.939224 2173 raft_consensus.cc:2743] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.940961 1628 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:52.942221 1779 tablet_bootstrap.cc:492] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:52.942188 2172 raft_consensus.cc:2798] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:52.942756 2172 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:52.942828 1779 ts_tablet_manager.cc:1397] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.207s user 0.011s sys 0.007s
I20250114 02:53:52.943183 2172 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:53:52.943598 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 45196.3us
I20250114 02:53:52.942757 1628 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:52.944561 1553 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:52.944957 1776 ts_tablet_manager.cc:1428] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.057s user 0.005s sys 0.002s
I20250114 02:53:52.945976 1628 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:52.914832 2191 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.947386 2216 raft_consensus.cc:2798] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250114 02:53:52.948088 1702 leader_election.cc:343] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:52.943027 2095 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.949126 2242 raft_consensus.cc:2798] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.949656 2242 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:52.950155 2242 raft_consensus.cc:3054] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.950356 2095 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:53:52.951056 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:53:52.952067 1776 tablet_bootstrap.cc:492] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:53:52.951596 2216 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:52.952728 2216 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.947788 2197 raft_consensus.cc:3054] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.961472 1779 raft_consensus.cc:357] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.969504 1779 raft_consensus.cc:383] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:52.959796 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:52.969874 1779 raft_consensus.cc:738] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:53:52.970347 1584 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.970594 1779 consensus_queue.cc:260] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.973013 1779 ts_tablet_manager.cc:1428] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.030s user 0.007s sys 0.001s
I20250114 02:53:52.960872 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:52.973861 1779 tablet_bootstrap.cc:492] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:53:52.974148 2216 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.959583 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:52.974515 2172 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.948711 2189 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:52.975848 2189 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.976478 2172 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:52.978178 2092 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:52.978464 2189 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:52.960505 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:52.980722 1729 raft_consensus.cc:3054] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.979489 1702 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:52.982290 1702 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:52.982980 2095 raft_consensus.cc:2743] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.983348 1702 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:52.983582 2095 raft_consensus.cc:2798] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.983927 2095 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:52.984264 2095 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.985155 1553 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:52.986711 2092 raft_consensus.cc:2798] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:52.960354 2232 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.987058 2092 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:52.987468 2092 raft_consensus.cc:3054] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:52.987408 2242 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.988173 2232 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:52.989300 2242 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:52.991417 2204 raft_consensus.cc:2743] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:52.994799 1553 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:52.995714 1702 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:52.996379 2248 raft_consensus.cc:2798] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:52.996760 2248 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:52.996701 2095 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:52.997144 2248 raft_consensus.cc:3054] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:52.958238 2106 raft_consensus.cc:695] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:52.998139 2204 raft_consensus.cc:2798] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.003228 2204 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:53.003623 2204 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.003656 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.004278 1581 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:53.001493 2244 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.007738 1586 raft_consensus.cc:1235] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.004758 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:53.008765 1733 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:53.011828 1585 raft_consensus.cc:1235] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.011616 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.011838 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.013327 2095 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.013583 1579 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.011597 2203 consensus_queue.cc:1046] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:53.014333 2230 raft_consensus.cc:3049] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.014675 2230 raft_consensus.cc:738] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.014689 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:53.014986 1734 raft_consensus.cc:1235] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.015314 1735 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.015540 2230 consensus_queue.cc:260] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.016052 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:52.997879 1584 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:53.018959 1580 raft_consensus.cc:1235] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.019748 2203 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:53.020200 2203 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.021979 2203 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.018061 2216 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.024061 1735 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:53.024716 2191 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.025596 2244 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.024147 2246 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:53.025903 1553 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:53.026027 1733 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:53.027575 1729 raft_consensus.cc:2463] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:53.032919 1779 tablet_bootstrap.cc:654] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:53.026752 1702 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:53.027753 2197 raft_consensus.cc:2743] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.030316 2246 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.030777 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:53.038060 1579 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:53.038291 2246 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.036203 2204 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.036101 1778 tablet_bootstrap.cc:492] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:53.039637 1778 ts_tablet_manager.cc:1397] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.173s user 0.015s sys 0.000s
I20250114 02:53:53.032838 2230 raft_consensus.cc:3054] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.042946 2191 raft_consensus.cc:2798] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.043397 1628 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:53.026372 2106 consensus_queue.cc:237] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.045959 2203 raft_consensus.cc:2798] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:53.047923 1778 raft_consensus.cc:357] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.048535 1778 raft_consensus.cc:383] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:53.048736 1778 raft_consensus.cc:738] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:53.049762 1702 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.050740 2229 consensus_queue.cc:1046] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:53.049186 1778 consensus_queue.cc:260] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.044384 2191 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:53.052947 2182 raft_consensus.cc:3049] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.053334 2191 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.053347 2182 raft_consensus.cc:738] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.054620 2182 consensus_queue.cc:260] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.055002 2092 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.055757 2182 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.057024 2092 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.055766 1581 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:53.050781 2203 raft_consensus.cc:695] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.058732 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.059363 1733 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:53.059954 1778 ts_tablet_manager.cc:1428] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250114 02:53:53.059314 2203 consensus_queue.cc:237] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.062379 2204 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.064733 1778 tablet_bootstrap.cc:492] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:53.044642 2246 raft_consensus.cc:2743] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.044764 2197 consensus_queue.cc:1046] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250114 02:53:53.069082 1776 tablet_bootstrap.cc:654] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
W20250114 02:53:53.069116 1628 leader_election.cc:343] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:53.069864 2232 raft_consensus.cc:3049] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.070729 1628 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:53.074412 1702 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.063832 2095 raft_consensus.cc:2743] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:53:53.076382 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 50531.3us
I20250114 02:53:53.076730 2175 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:53.077136 1553 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:53.077219 2175 raft_consensus.cc:695] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.078121 1628 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:53.079550 2174 raft_consensus.cc:2798] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:53.081326 2175 consensus_queue.cc:237] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.074337 2232 raft_consensus.cc:738] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.082741 2232 consensus_queue.cc:260] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.083564 2248 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.084050 2232 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.081661 2229 consensus_queue.cc:1046] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:53.089861 1628 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.086890 2174 raft_consensus.cc:695] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.093535 2174 consensus_queue.cc:237] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.087734 2095 raft_consensus.cc:2743] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:53.099323 1628 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.100195 1553 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:53.103706 1628 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.099493 2099 raft_consensus.cc:2798] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:53.105335 2231 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:53.104197 2106 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.104660 2229 raft_consensus.cc:2798] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:53.107018 2106 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:53.107199 2229 raft_consensus.cc:695] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.107431 2106 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.108028 2231 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:53.108461 2231 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:53.108105 2229 consensus_queue.cc:237] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.108654 2246 raft_consensus.cc:491] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:53.105988 2248 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.109135 2246 raft_consensus.cc:513] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.106309 2179 raft_consensus.cc:3049] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.115059 2179 raft_consensus.cc:738] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.115175 1579 raft_consensus.cc:1235] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.112282 2099 raft_consensus.cc:695] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.142374 2246 leader_election.cc:290] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.142776 2251 raft_consensus.cc:2798] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.144562 1583 raft_consensus.cc:1235] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.118634 2204 consensus_queue.cc:1046] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20250114 02:53:53.154731 2251 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:53.170841 2106 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:53.184646 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 47220.3us
I20250114 02:53:53.177739 2251 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.137090 2179 consensus_queue.cc:260] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.174127 2173 consensus_queue.cc:1046] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20250114 02:53:53.182796 2099 consensus_queue.cc:237] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:53.232473 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 53427.9us
I20250114 02:53:53.205494 2242 consensus_queue.cc:1035] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20250114 02:53:53.241194 2175 raft_consensus.cc:3049] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.241778 2175 raft_consensus.cc:738] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.234819 2179 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:53:53.249578 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 61300.8us
I20250114 02:53:53.235209 2191 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.250900 1553 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:53.255237 2248 raft_consensus.cc:2743] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.255584 1553 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:53.255760 2191 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.256356 2246 raft_consensus.cc:2798] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.256762 2246 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:53.257153 2246 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.250310 2259 raft_consensus.cc:3049] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.260416 2259 raft_consensus.cc:738] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:53.261186 2259 consensus_queue.cc:260] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.264986 1778 tablet_bootstrap.cc:654] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:53.252179 2178 consensus_queue.cc:1046] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20250114 02:53:53.253219 1728 raft_consensus.cc:1235] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.245287 2203 raft_consensus.cc:491] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:53.254256 1776 tablet_bootstrap.cc:492] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:53:53.268899 1776 ts_tablet_manager.cc:1397] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 0.317s user 0.009s sys 0.007s
W20250114 02:53:53.252280 2253 log.cc:926] Time spent T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20250114 02:53:53.267776 2175 consensus_queue.cc:260] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.271204 2178 raft_consensus.cc:3049] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.271595 2178 raft_consensus.cc:738] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.271903 2175 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.272038 2203 raft_consensus.cc:513] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.272534 2178 consensus_queue.cc:260] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.273495 1702 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.262445 2259 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.277376 1553 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:53.278214 1553 leader_election.cc:304] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:53.278314 2262 raft_consensus.cc:2798] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:53.281597 2260 consensus_queue.cc:1046] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250114 02:53:53.286077 2203 leader_election.cc:290] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.286414 2260 raft_consensus.cc:3049] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.286811 2260 raft_consensus.cc:738] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:53.275575 2106 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.287568 2106 raft_consensus.cc:2798] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:53.288087 2106 raft_consensus.cc:695] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:53.287652 2260 consensus_queue.cc:260] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.289090 2260 raft_consensus.cc:3054] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.288908 2106 consensus_queue.cc:237] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:53.273713 2176 consensus_queue.cc:1159] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b89550620e1d455d887be72232bb5454" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20250114 02:53:53.292587 2114 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:53.297065 2178 raft_consensus.cc:3054] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.315757 1702 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:53.316291 1776 raft_consensus.cc:357] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.316947 1776 raft_consensus.cc:383] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:53.317191 1776 raft_consensus.cc:738] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:53:53.317744 1776 consensus_queue.cc:260] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.322543 2255 raft_consensus.cc:2743] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.322670 2202 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
W20250114 02:53:53.325173 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 46174.2us
I20250114 02:53:53.310367 2262 raft_consensus.cc:695] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:53.327009 2262 consensus_queue.cc:237] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.314656 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.331427 1729 raft_consensus.cc:2463] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:53.308218 2114 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.332752 1628 leader_election.cc:304] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:53.333320 2114 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.335739 1702 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.334359 1729 raft_consensus.cc:1270] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:53.314827 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:53.340126 1779 tablet_bootstrap.cc:492] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:53:53.340541 1779 ts_tablet_manager.cc:1397] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 0.367s user 0.010s sys 0.002s
I20250114 02:53:53.340595 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:53.342675 1660 raft_consensus.cc:2388] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:53.342788 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:53.345871 1702 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:53.323395 2182 consensus_queue.cc:1046] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250114 02:53:53.346879 2259 raft_consensus.cc:2743] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:53.349148 2182 raft_consensus.cc:3049] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.349535 2182 raft_consensus.cc:738] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.349483 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.350257 1728 raft_consensus.cc:2463] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:53.341568 2114 raft_consensus.cc:2798] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:53.355571 1776 ts_tablet_manager.cc:1428] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.086s user 0.003s sys 0.002s
I20250114 02:53:53.350423 2182 consensus_queue.cc:260] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.351522 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:53.347788 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:53.343241 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:53.348865 1661 raft_consensus.cc:1235] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.353314 2231 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.353700 1579 raft_consensus.cc:3049] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.373768 1579 raft_consensus.cc:738] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:53.325587 2202 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.354085 2189 raft_consensus.cc:2798] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.377164 2189 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:53.377516 2189 raft_consensus.cc:3054] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.378176 2202 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.355434 2114 raft_consensus.cc:491] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:53.379400 2114 raft_consensus.cc:3054] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:53.374624 1628 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:53.375005 1702 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.376224 1579 consensus_queue.cc:260] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.380956 1779 raft_consensus.cc:357] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.381672 1779 raft_consensus.cc:383] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:53.381942 1779 raft_consensus.cc:738] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
W20250114 02:53:53.381842 1628 leader_election.cc:343] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:53.382449 1779 consensus_queue.cc:260] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.386158 2176 raft_consensus.cc:2798] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.404822 2176 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:53.405212 2176 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.405642 2259 raft_consensus.cc:2743] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:53.409042 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.409647 2263 raft_consensus.cc:2798] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.410955 1702 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:53.412143 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.412590 1779 ts_tablet_manager.cc:1428] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.072s user 0.005s sys 0.006s
I20250114 02:53:53.412519 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.412889 1581 raft_consensus.cc:1235] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.413179 1728 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:53.413401 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.414125 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:53.414736 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.414925 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:53.414753 1662 raft_consensus.cc:2463] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:53.405768 1579 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:53:53.416481 1628 leader_election.cc:343] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:53.415647 1655 raft_consensus.cc:2463] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:53.418208 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:53.419339 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.421963 1628 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:53.423940 1778 tablet_bootstrap.cc:492] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:53.424265 1778 ts_tablet_manager.cc:1397] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.360s user 0.011s sys 0.009s
W20250114 02:53:53.417285 1553 connection.cc:232] RPC call timeout handler was delayed by -1.4570571699999846s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1918s for a response
W20250114 02:53:53.425006 1553 connection.cc:232] RPC call timeout handler was delayed by -1.2149405669999851s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20510000000000003s for a response
I20250114 02:53:53.375078 2099 consensus_queue.cc:1035] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.036s
I20250114 02:53:53.426013 1729 raft_consensus.cc:1235] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.434732 2189 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.435528 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.435755 1660 raft_consensus.cc:2463] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:53.436101 1655 raft_consensus.cc:3054] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:53.436599 2189 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.423604 2182 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.410050 2263 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:53.431512 1778 raft_consensus.cc:357] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.434437 1661 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:53.442226 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.442728 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.423345 2244 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.425107 2202 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.422344 2229 consensus_queue.cc:1046] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20250114 02:53:53.407274 2246 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.443997 1702 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:53.444280 1778 raft_consensus.cc:383] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:53.445480 2244 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:53.447357 2229 raft_consensus.cc:3049] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.425144 2231 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.449301 1778 raft_consensus.cc:738] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:53.448182 2216 consensus_queue.cc:1046] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:53.445535 2202 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:53.456985 2202 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.454259 2229 raft_consensus.cc:738] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.452821 2251 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.453572 2244 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.459481 1553 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:53.459743 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.445039 2263 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.461841 2246 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.457255 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:53.464686 1728 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:53.464857 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:53.455735 1778 consensus_queue.cc:260] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.455993 1702 leader_election.cc:304] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.456508 2095 raft_consensus.cc:2798] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.450902 1484 catalog_manager.cc:5526] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:53.469333 1778 ts_tablet_manager.cc:1428] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.045s user 0.004s sys 0.002s
I20250114 02:53:53.465519 1654 raft_consensus.cc:2463] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:53.459218 2114 raft_consensus.cc:513] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.482326 1778 tablet_bootstrap.cc:492] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:53.478268 2251 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.478399 1579 raft_consensus.cc:1270] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:53.478593 2095 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:53.466055 2246 raft_consensus.cc:2743] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.478981 2229 consensus_queue.cc:260] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.484805 1702 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:53.485788 2259 raft_consensus.cc:2743] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:53.485996 2114 leader_election.cc:290] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.491004 2095 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.494624 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:53.495222 1729 raft_consensus.cc:3054] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:53.493749 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.498996 1583 raft_consensus.cc:3054] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:53.499931 2229 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.502686 1553 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:53.504844 1702 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.510349 1702 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.506521 2259 raft_consensus.cc:2798] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.504082 2244 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.515208 1581 raft_consensus.cc:1270] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:53.515604 2179 consensus_queue.cc:1035] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:53.507875 1653 raft_consensus.cc:2463] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:53.516707 1702 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:53.514134 2259 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:53.512449 1658 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:53.536964 1702 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:53.538267 2255 raft_consensus.cc:2798] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.538473 2191 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.538743 2255 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:53.538959 2191 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:53.539217 2255 raft_consensus.cc:3054] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.539363 2191 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.513989 2242 raft_consensus.cc:2743] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.540201 1702 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
W20250114 02:53:53.540547 1553 leader_election.cc:343] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:53.504520 2176 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.542753 1553 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:53.517969 1730 raft_consensus.cc:3049] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.514456 2189 raft_consensus.cc:2743] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.543905 1730 raft_consensus.cc:738] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:53.531548 1657 raft_consensus.cc:2463] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:53.544652 1735 raft_consensus.cc:1235] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.537462 2259 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.517060 2266 raft_consensus.cc:2743] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:53:53.546909 1702 leader_election.cc:343] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:53.547050 1553 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
W20250114 02:53:53.547503 1702 leader_election.cc:343] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:53:53.548070 1702 leader_election.cc:343] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:53:53.549088 2242 raft_consensus.cc:2798] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:53.549572 2242 raft_consensus.cc:695] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:53.553809 1778 tablet_bootstrap.cc:654] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:53.550429 2242 consensus_queue.cc:237] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.540171 1728 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:53.540782 2095 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.551920 2204 consensus_queue.cc:1035] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250114 02:53:53.524374 2202 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.557621 1553 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:53.565054 1735 raft_consensus.cc:1270] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250114 02:53:53.551060 1579 raft_consensus.cc:1235] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.560101 1583 raft_consensus.cc:2463] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:53.596016 1628 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.562024 1655 raft_consensus.cc:2463] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:53.568589 2191 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.579279 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.576239 2176 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.596119 1730 consensus_queue.cc:260] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.555869 1702 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:53.599627 2191 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.600771 2204 raft_consensus.cc:2798] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:53.601300 2204 raft_consensus.cc:695] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:53.603202 1628 leader_election.cc:304] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
W20250114 02:53:53.603478 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 47649us
I20250114 02:53:53.550948 2260 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.546497 2182 consensus_queue.cc:1035] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.016s
I20250114 02:53:53.575865 2095 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.558183 2246 raft_consensus.cc:2743] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.558641 2178 consensus_queue.cc:1046] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:53.566390 2263 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.580673 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:53.578759 1729 raft_consensus.cc:2463] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:53.594507 2244 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.598285 2262 raft_consensus.cc:2743] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.604518 1553 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:53.602257 2174 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:53.607214 2176 consensus_queue.cc:1035] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:53.631268 1658 raft_consensus.cc:1235] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.631865 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.629808 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:53.632164 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.633149 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.633215 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.634177 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.609041 2231 raft_consensus.cc:2798] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:53.608819 2260 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:53.574402 2255 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.602231 2204 consensus_queue.cc:237] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.578542 2259 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.599668 1730 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.639632 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.641006 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.641188 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:53.641827 1654 raft_consensus.cc:2388] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:53.642016 1734 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.642321 2231 raft_consensus.cc:695] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.630198 2263 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.648928 2174 raft_consensus.cc:695] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.628387 1731 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.663098 2114 consensus_queue.cc:1046] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:53.667150 1655 raft_consensus.cc:1235] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
W20250114 02:53:53.667886 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 63473.9us
I20250114 02:53:53.672111 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.660959 2174 consensus_queue.cc:237] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.643857 2246 raft_consensus.cc:2743] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.629982 2202 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.670055 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.692641 1661 raft_consensus.cc:2388] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:53.643268 2231 consensus_queue.cc:237] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.634429 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.639225 2260 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.693683 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.698194 1660 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:53:53.701388 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.701660 1736 raft_consensus.cc:1235] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.701725 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.703018 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.704288 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.705015 1735 raft_consensus.cc:2363] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2.
I20250114 02:53:53.705850 2259 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.706354 2259 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:53.706708 2179 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:53.706761 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.707018 2202 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:53.707248 2179 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.707500 2202 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.708026 1732 raft_consensus.cc:3049] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.708402 1732 raft_consensus.cc:738] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:53.709309 2182 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:53.709165 1732 consensus_queue.cc:260] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.709784 2182 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.710242 1732 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.710669 2178 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:53.711059 2175 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:53.711891 2182 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:53.712095 2276 log.cc:926] Time spent T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20250114 02:53:53.712364 2099 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:53.700506 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:53.716784 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.719962 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.712249 2229 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:53.720829 1730 raft_consensus.cc:1270] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:53.721035 1553 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:53.721655 1655 raft_consensus.cc:1235] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.727424 1731 raft_consensus.cc:2463] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:53.719213 2233 consensus_queue.cc:1035] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250114 02:53:53.731841 2251 raft_consensus.cc:2743] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.732115 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:53.733474 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:53.723788 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:53.709178 2179 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.725884 2099 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.731792 1585 raft_consensus.cc:1235] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.735384 1553 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:53.709190 2202 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.736152 1628 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.722465 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.742267 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.742738 1586 raft_consensus.cc:1235] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250114 02:53:53.722936 2178 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.706785 2259 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.731889 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.728318 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:53.722849 2175 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.754278 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:53.804033 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 67964.1us
I20250114 02:53:53.756618 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.805680 1661 raft_consensus.cc:2388] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:53.756047 1553 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:53.807518 2114 raft_consensus.cc:2798] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:53.811093 2114 raft_consensus.cc:695] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.754739 2248 raft_consensus.cc:2743] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.767192 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
W20250114 02:53:53.818655 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 62685.8us
I20250114 02:53:53.754102 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.819765 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:53.823698 2246 raft_consensus.cc:2743] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.821177 2259 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.821856 2178 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.827005 1778 tablet_bootstrap.cc:492] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:53.759091 1734 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:53.790882 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:53.767992 2255 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.788434 1658 raft_consensus.cc:2463] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:53:53.766767 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.827414 1778 ts_tablet_manager.cc:1397] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.346s user 0.014s sys 0.018s
I20250114 02:53:53.817777 2114 consensus_queue.cc:237] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.735783 2229 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.827322 1660 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
W20250114 02:53:53.829833 2274 log.cc:926] Time spent T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.242s user 0.002s sys 0.000s
I20250114 02:53:53.885378 1585 raft_consensus.cc:1235] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.824537 1580 raft_consensus.cc:1235] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:53.811745 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:53.825140 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.891803 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:53.895963 2279 log.cc:926] Time spent T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
W20250114 02:53:53.896088 2280 log.cc:926] Time spent T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20250114 02:53:53.896462 1730 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:53.897569 1582 raft_consensus.cc:3049] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.897953 1582 raft_consensus.cc:738] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:53.898552 2229 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.839839 1727 raft_consensus.cc:2463] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:53.899760 1732 raft_consensus.cc:1270] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:53.901332 2099 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.828684 2179 consensus_queue.cc:1035] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
I20250114 02:53:53.901923 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.904330 1628 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:53.906214 2099 raft_consensus.cc:2743] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.909791 1553 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:53.912014 1628 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.887034 2246 consensus_queue.cc:1035] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20250114 02:53:53.912963 1628 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.915463 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:53.916589 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.917393 1727 raft_consensus.cc:2388] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:53.912389 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:53.918623 2202 consensus_queue.cc:1046] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:53:53.920688 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:53.921262 1727 raft_consensus.cc:3054] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:53:53.924757 2277 log.cc:926] Time spent T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.135s user 0.001s sys 0.000s
I20250114 02:53:53.927639 1628 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:53.908597 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.931739 1778 raft_consensus.cc:357] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.932780 1778 raft_consensus.cc:383] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:53.933152 1778 raft_consensus.cc:738] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:53:53.934010 1778 consensus_queue.cc:260] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.909117 2231 consensus_queue.cc:1035] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20250114 02:53:53.936138 1778 ts_tablet_manager.cc:1428] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.050s user 0.005s sys 0.001s
I20250114 02:53:53.936519 2216 raft_consensus.cc:3049] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.936784 2216 raft_consensus.cc:738] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.937328 2216 consensus_queue.cc:260] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.938071 2216 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.939155 2179 raft_consensus.cc:2743] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.940287 1628 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:53.928303 2262 raft_consensus.cc:2798] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:53.927415 2233 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:53.910390 2106 consensus_queue.cc:1046] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250114 02:53:53.930079 1584 raft_consensus.cc:1270] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:53.929131 2197 raft_consensus.cc:2798] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:53.906229 1582 consensus_queue.cc:260] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.951278 1582 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.941102 2176 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:53.948251 1702 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.949234 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:53.949956 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:53.950539 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.942319 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:53.930629 2178 consensus_queue.cc:1035] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20250114 02:53:53.944150 2262 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:53.955232 2233 raft_consensus.cc:695] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:53.946501 2175 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:53.950332 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:53.967046 2259 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:53.995649 1734 raft_consensus.cc:1235] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:54.040807 2092 raft_consensus.cc:3049] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:54.041298 2092 raft_consensus.cc:738] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:54.040920 2259 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.987854 1733 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:54.042960 2259 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:54.042193 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:54.015923 2260 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:53.967155 1702 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
W20250114 02:53:54.056145 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 89358.9us
I20250114 02:53:54.057408 1702 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:54.019403 1778 tablet_bootstrap.cc:492] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:53:53.994503 1735 raft_consensus.cc:2388] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:54.059628 1702 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:54.054549 1628 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:53.999138 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:54.016760 1583 raft_consensus.cc:3049] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.947700 2197 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:53.991555 2262 raft_consensus.cc:3054] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:53.967389 2255 raft_consensus.cc:2743] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:53.995486 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:54.050853 2174 raft_consensus.cc:3049] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:53.999517 1727 raft_consensus.cc:2463] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:54.010360 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:54.015342 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:54.073771 1583 raft_consensus.cc:738] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:54.075677 2174 raft_consensus.cc:738] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
W20250114 02:53:54.113251 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 53719.6us
I20250114 02:53:54.073935 2197 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:53:54.112540 2289 log.cc:926] Time spent T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
W20250114 02:53:54.125483 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 215751us
I20250114 02:53:54.013000 2182 consensus_queue.cc:1035] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20250114 02:53:54.073741 1582 raft_consensus.cc:1270] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250114 02:53:54.042117 2092 consensus_queue.cc:260] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.075655 2179 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:54.071089 2204 raft_consensus.cc:2743] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:54.280721 2268 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:54.073662 1731 raft_consensus.cc:2463] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 0.
I20250114 02:53:54.067813 1734 raft_consensus.cc:2463] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:54.054236 2175 consensus_queue.cc:1046] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.104s
I20250114 02:53:54.068120 2259 raft_consensus.cc:2743] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:54.338321 1736 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:54.339412 2260 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:53.995016 2233 consensus_queue.cc:237] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.346632 1553 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:54.337195 2246 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:54.340234 2251 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:54.374624 1553 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:54.350672 1582 pending_rounds.cc:77] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 0
I20250114 02:53:54.357908 2268 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.353993 2179 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:54.346138 1732 raft_consensus.cc:1212] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250114 02:53:54.359082 2092 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:54.360369 1731 raft_consensus.cc:1235] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:54.362735 2174 consensus_queue.cc:260] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.364311 1778 tablet_bootstrap.cc:654] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:54.364692 1583 consensus_queue.cc:260] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.367050 2262 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.350837 2176 raft_consensus.cc:2743] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:54.349228 2178 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:54.374819 1730 raft_consensus.cc:1235] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:54.342417 2263 raft_consensus.cc:2798] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:54.354661 1727 raft_consensus.cc:1270] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250114 02:53:54.377686 2248 raft_consensus.cc:2798] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:54.381393 2182 raft_consensus.cc:3049] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:54.393500 2179 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:54.388149 1582 pending_rounds.cc:99] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250114 02:53:54.410899 1732 raft_consensus.cc:1235] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250114 02:53:54.421658 2114 consensus_queue.cc:1035] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:54.387044 2251 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.418150 1583 raft_consensus.cc:3054] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:54.395915 2246 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.424050 2248 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:54.418975 2263 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:54.427486 2174 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:54.442800 2263 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:54.444689 1732 raft_consensus.cc:1235] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:54.430516 1582 raft_consensus.cc:2883] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:53:54.447487 2248 raft_consensus.cc:3054] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:54.448261 2251 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:54.451236 1727 raft_consensus.cc:1235] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:54.423781 2182 raft_consensus.cc:738] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:54.453722 2182 consensus_queue.cc:260] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.459822 2268 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:54.461377 2176 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:54.462098 2176 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.464540 2176 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:54.427242 1729 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.010s sys 0.012s
W20250114 02:53:54.468478 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 93969.3us
I20250114 02:53:54.469875 1553 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:54.418752 2178 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:54.497910 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 153541us
I20250114 02:53:54.436198 1734 raft_consensus.cc:1235] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:54.474817 1553 leader_election.cc:304] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
W20250114 02:53:54.469388 1729 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s user 0.011s sys 0.012s
I20250114 02:53:54.436318 2106 raft_consensus.cc:2798] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:54.473789 2176 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:54.506572 2268 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election won for term 2
W20250114 02:53:54.527740 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 53010.7us
I20250114 02:53:54.533201 1702 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:54.532444 2106 raft_consensus.cc:695] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:54.547348 2106 consensus_queue.cc:237] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.547343 2246 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:54.528540 1731 raft_consensus.cc:1235] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:54.555174 2179 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.557327 2179 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:54.557379 2268 raft_consensus.cc:695] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:54.558549 2268 consensus_queue.cc:237] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.562995 2178 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:54.564188 2248 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.564556 2174 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:54.564999 2174 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.566001 2248 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:54.566793 2174 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:54.528649 2197 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.567121 1583 raft_consensus.cc:1270] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250114 02:53:54.567559 1583 pending_rounds.cc:77] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 0
I20250114 02:53:54.567835 1583 pending_rounds.cc:99] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250114 02:53:54.568138 1583 raft_consensus.cc:2883] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:53:54.568732 2197 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:54.569380 2298 raft_consensus.cc:2743] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:54.537290 1553 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:54.571569 2092 raft_consensus.cc:2798] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:54.571745 1553 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:54.572074 2092 raft_consensus.cc:695] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:54.544265 2176 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.572963 2092 consensus_queue.cc:237] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.573886 2176 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:54.532948 2251 raft_consensus.cc:2798] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:54.574848 2251 raft_consensus.cc:491] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:54.575268 2251 raft_consensus.cc:3054] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:54.576438 2284 raft_consensus.cc:2798] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:54.576995 2284 raft_consensus.cc:695] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:54.577955 2284 consensus_queue.cc:237] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.582387 1553 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:54.562131 2263 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.587652 2263 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:54.596104 2246 raft_consensus.cc:2798] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:54.596730 2246 raft_consensus.cc:695] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:54.599570 1702 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:54.601998 2298 consensus_queue.cc:1035] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:54.597530 2246 consensus_queue.cc:237] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.618844 2233 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:54.619458 2233 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.621994 2233 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:54.592988 2251 raft_consensus.cc:513] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.624927 2251 leader_election.cc:290] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:54.595224 1553 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:54.632018 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:54.632839 1656 raft_consensus.cc:2388] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:54.633702 2298 raft_consensus.cc:2798] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:53:54.634107 2298 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:54.634510 2298 raft_consensus.cc:3054] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:54.634338 1702 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:54.635499 2307 raft_consensus.cc:2743] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:54.644305 2302 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:54.644968 2302 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.647543 2302 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:54.630079 2248 raft_consensus.cc:2798] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:54.657792 1702 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
W20250114 02:53:54.660838 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 242030us
I20250114 02:53:54.675380 1702 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:54.680601 1702 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:54.683722 1586 raft_consensus.cc:3049] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:54.684074 1586 raft_consensus.cc:738] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:54.684444 1702 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:54.685433 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:54.686380 1732 raft_consensus.cc:2388] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:54.688264 1778 tablet_bootstrap.cc:492] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:53:54.688814 1778 ts_tablet_manager.cc:1397] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 0.670s user 0.032s sys 0.042s
I20250114 02:53:54.689882 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:54.683965 2263 raft_consensus.cc:2798] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:54.694671 2260 raft_consensus.cc:2798] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:54.695029 2263 raft_consensus.cc:695] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:54.695673 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:54.695844 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:54.698163 1586 consensus_queue.cc:260] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.720312 1586 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:54.687614 2307 raft_consensus.cc:2798] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:54.694691 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:54.686991 2300 raft_consensus.cc:2798] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:54.687232 2248 raft_consensus.cc:695] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:54.695933 2263 consensus_queue.cc:237] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.808175 1586 raft_consensus.cc:1270] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:54.759495 1628 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:54.759485 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:55.010202 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 250824us
I20250114 02:53:55.011723 1628 leader_election.cc:304] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:54.722667 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:55.012432 1734 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:53:55.008693 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 321077us
I20250114 02:53:54.765497 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:55.045495 1727 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:53:54.961410 2274 log.cc:926] Time spent T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20250114 02:53:55.057817 1734 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:55.075805 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:54.846240 2304 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:54.760325 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:54.803092 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:55.045132 2248 consensus_queue.cc:237] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.046146 2284 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.011148 2204 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.046124 2297 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:55.089756 2304 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:54.746162 2174 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.002177 2178 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:54.992141 2233 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:55.089617 1732 raft_consensus.cc:1270] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:55.125391 2174 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.151722 2304 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.151235 2179 raft_consensus.cc:2798] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250114 02:53:55.128573 2293 log.cc:926] Time spent T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20250114 02:53:54.964591 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:55.125862 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:54.746387 2197 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
W20250114 02:53:55.129238 2313 log.cc:926] Time spent T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
I20250114 02:53:55.170398 1778 raft_consensus.cc:357] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.195755 1778 raft_consensus.cc:383] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:55.196121 1778 raft_consensus.cc:738] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
W20250114 02:53:55.129807 2306 log.cc:926] Time spent T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.283s user 0.000s sys 0.001s
I20250114 02:53:55.090765 2315 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.106544 2262 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.107488 1584 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:55.022091 2299 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:54.773799 2260 raft_consensus.cc:695] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
W20250114 02:53:55.162312 2298 consensus_meta.cc:287] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.526s user 0.013s sys 0.003s
W20250114 02:53:55.191627 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 139886us
I20250114 02:53:55.125732 2204 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.109118 2284 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.197000 1778 consensus_queue.cc:260] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.192876 2179 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:55.171981 2178 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.196796 2175 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:55.226447 1778 ts_tablet_manager.cc:1428] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 0.537s user 0.016s sys 0.004s
I20250114 02:53:55.195820 2304 consensus_queue.cc:1035] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20250114 02:53:55.193573 2233 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.201287 2300 raft_consensus.cc:695] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:55.196259 2197 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.216200 2315 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.201092 2317 consensus_queue.cc:1046] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
W20250114 02:53:55.226895 1778 ts_tablet_manager.cc:1460] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd: Tablet startup took 537ms
I20250114 02:53:55.260730 1656 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:55.244236 1733 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
W20250114 02:53:55.313205 1778 ts_tablet_manager.cc:1462] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd: Trace:
0114 02:53:41.921990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:42.660746 (+738756us) service_pool.cc:225] Handling call
0114 02:53:42.703355 (+ 42609us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0114 02:53:42.706094 (+ 2739us) ts_tablet_manager.cc:676] Creating new metadata...
0114 02:53:42.829556 (+123462us) tablet_metadata.cc:641] Metadata flushed
0114 02:53:42.893146 (+ 63590us) tablet_replica.cc:206] Creating consensus instance
0114 02:53:42.893884 (+ 738us) inbound_call.cc:173] Queueing success response
0114 02:53:54.019002 (+11125118us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0114 02:53:54.689206 (+670204us) ts_tablet_manager.cc:1429] Starting tablet replica
0114 02:53:55.168233 (+479027us) tablet_replica.cc:260] Starting instrumentation
0114 02:53:55.168898 (+ 665us) tablet_replica.cc:279] Starting consensus
Metrics: {"mutex_wait_us":3601,"raft.queue_time_us":14531,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":14642,"tablet-open.queue_time_us":11125499}
I20250114 02:53:55.257813 2259 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:55.315544 2259 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.317059 2297 raft_consensus.cc:2743] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:55.317641 2259 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:55.265187 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 208) took 1373 ms. Trace:
I20250114 02:53:55.220518 2179 raft_consensus.cc:3054] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:53:55.259686 2316 log.cc:926] Time spent T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20250114 02:53:55.217774 2298 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:55.326401 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 312408us
I20250114 02:53:55.267365 2284 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.339543 1702 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:55.261521 2204 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:55.255762 2178 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.253157 2246 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:55.269063 2174 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.282641 2327 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:55.342886 1702 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:55.253931 2326 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:55.270439 1727 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:55.247803 2322 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.351462 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:55.377292 1586 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:55.378309 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:55.340724 2106 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:55.227572 2175 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:55.391196 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 47832.4us
I20250114 02:53:55.324980 2114 consensus_queue.cc:1046] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
W20250114 02:53:55.414731 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 79061.7us
I20250114 02:53:55.374830 2246 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.327883 2302 consensus_queue.cc:1035] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.055s
I20250114 02:53:55.218230 2299 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.417896 2324 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.418368 2324 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.418855 2299 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.416683 2106 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.419553 2175 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.421066 2106 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.361330 2319 raft_consensus.cc:2798] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:55.265112 2321 consensus_queue.cc:1046] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.116s
I20250114 02:53:55.416755 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:55.321229 2297 consensus_queue.cc:1046] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250114 02:53:55.375257 2317 raft_consensus.cc:2798] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:55.374533 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:55.318523 1730 rpcz_store.cc:276] 0114 02:53:53.892982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:54.119928 (+226946us) service_pool.cc:225] Handling call
0114 02:53:54.120327 (+ 399us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:54.424763 (+304436us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:55.264932 (+840169us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b54002aec54. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:53:55.265111 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42547968}
I20250114 02:53:55.423233 1586 raft_consensus.cc:2463] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:53:55.265939 2260 consensus_queue.cc:237] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.378707 2251 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:55.414992 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:55.432291 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:55.468776 1553 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:55.424238 2304 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.471788 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:55.414495 2323 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.472187 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:55.472788 1586 raft_consensus.cc:2463] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:53:55.375521 2174 consensus_queue.cc:1035] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.055s
I20250114 02:53:55.431453 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:55.427191 2259 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:55.431718 2106 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.478488 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:55.432832 2317 raft_consensus.cc:695] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:55.477705 2175 raft_consensus.cc:3049] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:55.462186 2332 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.537338 1553 leader_election.cc:400] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2.
I20250114 02:53:55.378608 2327 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.475595 2284 raft_consensus.cc:2798] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:55.475730 2299 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.480720 2204 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.461621 2251 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.380532 2326 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.475193 2304 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.485196 2175 raft_consensus.cc:738] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:55.491523 2324 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.484536 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:55.422525 2322 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.539587 2248 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:55.541067 2262 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:55.481019 2259 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:55.548349 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 66631.4us
I20250114 02:53:55.548061 2334 raft_consensus.cc:3049] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:55.545352 2319 raft_consensus.cc:695] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:55.544931 2284 raft_consensus.cc:695] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:55.548012 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:55.481305 2106 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.543964 2297 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:55.545385 2233 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.564277 1553 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:55.480511 2323 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.546000 2263 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.543593 2197 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.552076 2298 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:55.551187 2321 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:53:55.576892 2329 log.cc:926] Time spent T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.185s user 0.000s sys 0.001s
I20250114 02:53:55.577638 2251 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.552047 2302 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:55.571502 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:55.556172 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 203) took 1827 ms. Trace:
I20250114 02:53:55.580345 1702 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:55.578045 2175 consensus_queue.cc:260] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.425101 2300 consensus_queue.cc:237] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.582262 1553 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:55.544787 2299 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.563315 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 268) took 1846 ms. Trace:
I20250114 02:53:55.585204 1661 raft_consensus.cc:1235] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:55.565246 2262 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.564277 2334 raft_consensus.cc:738] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:55.577680 2304 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.555584 2315 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:55.588346 2284 consensus_queue.cc:237] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.546398 2204 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.568696 2248 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.588647 2327 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.587272 2263 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.590514 2321 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.592649 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:55.540374 2332 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.588204 2319 consensus_queue.cc:237] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.584625 2297 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.572893 2317 consensus_queue.cc:237] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.596797 2251 raft_consensus.cc:2743] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250114 02:53:55.787645 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 192049us
W20250114 02:53:55.714989 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 298202us
I20250114 02:53:55.793902 1628 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:55.794406 1553 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
W20250114 02:53:55.787871 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 207603us
I20250114 02:53:55.673556 1661 raft_consensus.cc:1235] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:55.673981 1658 raft_consensus.cc:2388] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
W20250114 02:53:55.795259 1702 leader_election.cc:343] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:53:55.776377 2338 log.cc:926] Time spent T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.190s user 0.000s sys 0.000s
I20250114 02:53:55.819296 1702 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:55.819936 2106 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.820667 2259 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:55.824360 1628 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:55.668387 1653 raft_consensus.cc:1235] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:55.824646 1702 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:55.672390 2303 raft_consensus.cc:2743] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:55.823851 2298 raft_consensus.cc:2743] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:55.826386 2114 consensus_queue.cc:1035] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:53:55.674242 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:55.674705 2343 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:55.817567 2327 raft_consensus.cc:2798] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:55.672433 2324 raft_consensus.cc:491] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.638243 2246 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.787129 1654 raft_consensus.cc:1212] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250114 02:53:55.595386 1662 raft_consensus.cc:2463] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:53:55.832620 1702 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:55.695854 2326 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:55.866281 1702 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:55.818077 2251 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.835757 2259 raft_consensus.cc:2743] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:55.796804 2323 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:55.797757 2309 log.cc:926] Time spent T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.742s user 0.006s sys 0.000s
I20250114 02:53:55.592363 2302 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.624048 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:55.826326 2233 consensus_queue.cc:1035] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:55.793279 2175 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:55.826727 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 318) took 1135 ms. Trace:
I20250114 02:53:55.723299 2197 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:55.671651 2341 raft_consensus.cc:2743] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:55.725929 2322 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.795946 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 289) took 1119 ms. Trace:
I20250114 02:53:55.799937 2299 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.666814 2330 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:55.799582 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:55.796972 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:55.864617 1654 raft_consensus.cc:1235] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250114 02:53:55.864519 2324 raft_consensus.cc:513] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.672184 2179 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.871942 2260 raft_consensus.cc:2743] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:55.855614 2106 raft_consensus.cc:2758] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250114 02:53:55.868566 1702 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:55.873206 2259 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:55.865269 2229 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:55.869567 2326 raft_consensus.cc:2743] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:55.864636 2242 consensus_queue.cc:1046] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20250114 02:53:55.910671 1660 raft_consensus.cc:1235] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
W20250114 02:53:55.912397 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 43908.7us
I20250114 02:53:55.904632 1655 raft_consensus.cc:2388] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:55.916639 1702 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:55.899097 2246 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:55.908098 1579 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:55.870640 2251 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.925585 2324 leader_election.cc:290] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.925467 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:55.881647 2197 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.914865 2298 raft_consensus.cc:2743] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:55.934347 1702 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
W20250114 02:53:55.937342 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 66306.1us
I20250114 02:53:55.881019 1580 rpcz_store.cc:276] 0114 02:53:54.676174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:54.680024 (+ 3850us) service_pool.cc:225] Handling call
0114 02:53:55.588956 (+908932us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315ed0e 0000000000353db9 00007fbceafbea50 00007fbceb084814 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:53:55.795771 (+206815us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7452416}
I20250114 02:53:55.937753 2328 raft_consensus.cc:2743] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:55.872742 1733 rpcz_store.cc:276] 0114 02:53:54.691540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:54.746189 (+ 54649us) service_pool.cc:225] Handling call
0114 02:53:55.826496 (+1080307us) spinlock_profiling.cc:243] Waited 31.5 ms on lock 0x7b540007fb54. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:55.826644 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":72541952}
I20250114 02:53:55.912164 2321 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.926410 1657 raft_consensus.cc:2388] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:55.917865 2327 raft_consensus.cc:695] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:55.787192 2334 consensus_queue.cc:260] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.918763 2348 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:55.924222 2229 raft_consensus.cc:695] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:55.926368 2106 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.943403 1628 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
W20250114 02:53:55.926829 1579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 293) took 1246 ms (client timeout 1533 ms). Trace:
I20250114 02:53:55.871856 2343 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.947029 1702 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:55.926508 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 298) took 1981 ms. Trace:
I20250114 02:53:55.951921 1628 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:55.924664 2246 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.926820 2259 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.938508 1655 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:55.940755 2297 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.942065 2248 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.957360 1628 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:55.961512 2346 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:55.962917 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 285) took 1590 ms. Trace:
I20250114 02:53:55.962360 2092 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.959504 2334 raft_consensus.cc:3054] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:55.949896 2204 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:55.947822 2263 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:55.959059 2317 raft_consensus.cc:2798] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:55.970266 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:55.952543 2321 raft_consensus.cc:2743] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:55.949455 2326 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:55.966331 1585 rpcz_store.cc:276] 0114 02:53:54.372903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:54.373031 (+ 128us) service_pool.cc:225] Handling call
0114 02:53:54.377142 (+ 4111us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:53:54.523174 (+146032us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:54.523220 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:53:54.533948 (+ 10728us) log.cc:844] Serialized 26 byte log entry
0114 02:53:54.728818 (+194870us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:54.728855 (+ 37us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:54.728930 (+ 75us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:53:55.786628 (+1057698us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b440002494c. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce30ad7c5 00007fbce30ae1e2 00007fbce5d63da1 00007fbce5d5f68a 00007fbce5d534ae 00007fbce5d527f6 00007fbce5d52418 00007fbce5d5bdf1 00007fbce5d5bda9 00007fbce5d5bd39
0114 02:53:55.949507 (+162879us) raft_consensus.cc:1680] finished
0114 02:53:55.949527 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:55.962838 (+ 13311us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3976,"raft.queue_time_us":47741,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":5339,"spinlock_wait_cycles":3473792,"thread_start_us":290776,"threads_started":2,"wal-append.queue_time_us":465086}
I20250114 02:53:55.969401 2092 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.924384 2330 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.947451 2298 consensus_queue.cc:1046] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20250114 02:53:55.925513 2352 consensus_queue.cc:1035] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.046s
I20250114 02:53:55.978003 1587 raft_consensus.cc:3049] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:56.001744 1587 raft_consensus.cc:738] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:55.959684 2303 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:53:55.949476 1579 rpcz_store.cc:269] 0114 02:53:54.677307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:54.680202 (+ 2895us) service_pool.cc:225] Handling call
0114 02:53:55.796845 (+1116643us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315ed0e 0000000000353db9 00007fbceafbea50 00007fbceb084814 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:53:55.923870 (+127025us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7400704}
I20250114 02:53:56.004077 2246 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.003964 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:55.592383 1728 rpcz_store.cc:276] 0114 02:53:53.727454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:54.001442 (+273988us) service_pool.cc:225] Handling call
0114 02:53:54.001732 (+ 290us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:54.001963 (+ 231us) raft_consensus.cc:1529] Early marking committed up to index 1
0114 02:53:54.052995 (+ 51032us) log.cc:844] Serialized 14 byte log entry
0114 02:53:54.419646 (+366651us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b18001d0e00. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce3169ec6 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce319c949 00007fbce319fcc1
0114 02:53:55.045716 (+626070us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b18001d0e00. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce319c7e8 00007fbce319fcc1 00007fbce319fc79 00007fbce319fc09 00007fbce319fbd1 00007fbce319eecd 00007fbcec1d9234 00007fbcec1d9069 00007fbce317efe6
0114 02:53:55.483565 (+437849us) spinlock_profiling.cc:243] Waited 5.39 ms on lock 0x7b18001d0e00. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce5d6008c 00007fbce317ebfc 00007fbcebd907f9 00007fbce319ba6b 00007fbce3199e6f 00007fbce319bfd2 00007fbce5d51e4b 00007fbce5d57913 00007fbce5d57e2b
0114 02:53:55.540010 (+ 56445us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:55.541496 (+ 1486us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:55.545113 (+ 3617us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:55.548432 (+ 3319us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:55.554436 (+ 6004us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8936,"prepare.queue_time_us":50968,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":46,"raft.queue_time_us":57975,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":3765,"spinlock_wait_cycles":38873856,"thread_start_us":58621,"threads_started":2,"wal-append.queue_time_us":8574}
I20250114 02:53:55.950822 2106 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:55.970402 2297 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:55.977986 1733 raft_consensus.cc:1270] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:55.978456 2248 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:55.977584 2318 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:55.977854 2340 raft_consensus.cc:2743] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:55.978330 2317 raft_consensus.cc:695] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:55.983850 2179 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.969656 2343 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:55.596602 1581 rpcz_store.cc:276] 0114 02:53:53.717371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:53.822868 (+105497us) service_pool.cc:225] Handling call
0114 02:53:53.823247 (+ 379us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 02:53:53.891433 (+ 68186us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:53.892867 (+ 1434us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 02:53:53.930491 (+ 37624us) log.cc:844] Serialized 52 byte log entry
0114 02:53:54.042930 (+112439us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:55.423425 (+1380495us) spinlock_profiling.cc:243] Waited 7.21 ms on lock 0x7b1800098080. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce5d6008c 00007fbce317ebfc 00007fbcebd907f9 00007fbce319ba6b 00007fbce3199e6f 00007fbce3199305 00007fbce9aaa107 00007fbce60df8f7 00007fbce60ef00f
0114 02:53:55.424970 (+ 1545us) log.cc:844] Serialized 14 byte log entry
0114 02:53:55.485615 (+ 60645us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:55.540307 (+ 54692us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:53:55.543720 (+ 3413us) raft_consensus.cc:1680] finished
0114 02:53:55.549818 (+ 6098us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:55.558044 (+ 8226us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72145,"prepare.queue_time_us":298695,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":67235,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":54941,"spinlock_wait_cycles":16574336,"thread_start_us":451481,"threads_started":3,"wal-append.queue_time_us":248495,"wal-append.run_cpu_time_us":2013,"wal-append.run_wall_time_us":1013762}
I20250114 02:53:56.021678 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.021987 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.022888 2259 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:56.023108 2197 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.023545 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.024407 2092 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.972515 2260 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:56.053102 1586 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:55.986124 2346 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:55.951169 2348 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.018141 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.012226 2302 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.025866 1587 consensus_queue.cc:260] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.114095 1587 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.006105 2326 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.090858 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.053141 2353 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.110715 2346 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.024989 2303 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.134858 2350 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:56.135304 2354 consensus_queue.cc:1035] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.180s
I20250114 02:53:56.135600 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 287) took 1460 ms. Trace:
I20250114 02:53:56.016677 2332 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:55.952723 1736 rpcz_store.cc:276] 0114 02:53:53.944972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:54.374109 (+429137us) service_pool.cc:225] Handling call
0114 02:53:54.376100 (+ 1991us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:53:54.376511 (+ 411us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:54.376553 (+ 42us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:53:55.546997 (+1170444us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b18001a9620. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce3169ec6 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce4ab6741 00007fbce4ac5d72 00007fbce6111141 00007fbce60d558a
0114 02:53:55.800120 (+253123us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b2c0003ccf8. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce4ab6741 00007fbce4ac5d72 00007fbce6111141 00007fbce60d558a 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782
0114 02:53:55.828213 (+ 28093us) log.cc:844] Serialized 26 byte log entry
0114 02:53:55.902634 (+ 74421us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:55.908746 (+ 6112us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:55.912120 (+ 3374us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:53:55.916051 (+ 3931us) raft_consensus.cc:1680] finished
0114 02:53:55.919166 (+ 3115us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:55.923597 (+ 4431us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4843,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":10114,"spinlock_wait_cycles":6838400,"thread_start_us":51426,"threads_started":1,"wal-append.queue_time_us":78567}
I20250114 02:53:56.101464 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:56.030071 2248 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.113576 2179 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.090543 2233 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.117069 2197 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.136111 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:56.145592 2330 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:56.117630 2251 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.104182 1730 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:56.113065 1731 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:56.014321 2229 consensus_queue.cc:237] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.014493 2327 consensus_queue.cc:237] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.148797 2332 raft_consensus.cc:491] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:56.162827 2332 raft_consensus.cc:513] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.137571 2352 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.112079 2260 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.143790 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:56.141999 1727 raft_consensus.cc:2463] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:56.117899 2318 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.153892 1579 raft_consensus.cc:2463] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:56.156180 2242 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:56.189162 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:56.154871 2233 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.153303 2179 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.176759 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.156925 2197 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.189302 2352 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.189167 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:56.193185 2255 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:56.217303 2255 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.219172 2255 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:56.143867 1583 rpcz_store.cc:276] 0114 02:53:54.675547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:54.680298 (+ 4751us) service_pool.cc:225] Handling call
0114 02:53:56.117246 (+1436948us) spinlock_profiling.cc:243] Waited 2.48 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:53:56.135016 (+ 17770us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:56.135203 (+ 187us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:56.135242 (+ 39us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:56.135302 (+ 60us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:56.135337 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:56.135418 (+ 81us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:56.135539 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5715328}
I20250114 02:53:56.219874 2197 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.220193 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:56.220533 2197 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.220862 1583 raft_consensus.cc:2463] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:56.221228 2297 raft_consensus.cc:2743] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:56.221139 2197 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.221936 2297 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.194905 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.216781 2233 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.221781 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:56.212646 2332 leader_election.cc:290] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.224721 2318 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:56.205680 2106 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:56.227824 2358 log.cc:926] Time spent T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.084s user 0.001s sys 0.001s
I20250114 02:53:56.143121 2353 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.229372 2248 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.222388 2297 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.192054 2242 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.187923 2303 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.210709 1587 raft_consensus.cc:1270] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:56.233358 1583 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:56.234134 2297 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.234093 2353 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.237778 2179 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.225230 2348 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.229964 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:56.238792 2242 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.238351 1581 raft_consensus.cc:2463] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:56.240499 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:56.245914 2355 log.cc:926] Time spent T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20250114 02:53:56.235643 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.240754 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:56.239908 2260 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625)
I20250114 02:53:56.239794 2326 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:56.251435 1580 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
W20250114 02:53:56.276301 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 329304us
I20250114 02:53:56.158859 2317 consensus_queue.cc:237] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.277745 1702 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:56.274883 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:56.241318 2352 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.291569 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.310525 1661 raft_consensus.cc:2388] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:56.314162 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:56.332273 1661 raft_consensus.cc:3054] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.153682 2350 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.284508 1655 raft_consensus.cc:1235] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:56.285002 2260 raft_consensus.cc:2798] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:56.291379 2331 raft_consensus.cc:3049] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:56.285357 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.279019 1578 raft_consensus.cc:2463] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:56.297048 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.283190 2197 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.285260 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.293469 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.347510 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:56.347658 2357 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.348142 1728 raft_consensus.cc:2388] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:56.354598 1661 raft_consensus.cc:2463] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:56.285512 2340 consensus_queue.cc:1035] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.068s
I20250114 02:53:56.368396 1585 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:56.355917 1657 raft_consensus.cc:3054] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:56.357911 2182 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.344959 2331 raft_consensus.cc:738] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
W20250114 02:53:56.310338 2307 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.225s user 0.023s sys 0.024s
I20250114 02:53:56.365607 2318 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.421383 2197 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.366119 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.422060 2197 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.423048 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.367599 2336 raft_consensus.cc:3049] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:56.340480 2352 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:56.342931 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.364894 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:56.344645 2260 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:56.421171 2318 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.421463 2350 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.366665 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.450891 1733 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:56.451730 2318 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:56.451887 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.447863 1661 raft_consensus.cc:3054] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:56.451730 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.453073 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:56.453656 1653 raft_consensus.cc:2463] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:56.365242 1662 raft_consensus.cc:2463] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:56.405802 2357 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.393332 2346 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.447548 2260 raft_consensus.cc:3054] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:56.447866 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:56.448788 2336 raft_consensus.cc:738] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:56.449311 2352 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.453711 1733 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.368822 1735 raft_consensus.cc:2463] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:56.454725 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.461127 1661 raft_consensus.cc:2463] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
W20250114 02:53:56.427371 2307 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.330s user 0.023s sys 0.024s
I20250114 02:53:56.478988 2197 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.485416 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.488731 2307 raft_consensus.cc:695] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:56.490031 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.492053 1661 raft_consensus.cc:2463] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:56.493981 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:56.494076 2336 consensus_queue.cc:260] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.496387 2336 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.495604 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.496464 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.489264 1728 raft_consensus.cc:3054] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:56.498543 1661 raft_consensus.cc:2463] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:56.498037 1659 raft_consensus.cc:1235] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:56.454555 2331 consensus_queue.cc:260] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.502770 1653 raft_consensus.cc:2388] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:56.504101 2352 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.504602 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.509871 2331 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.510694 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:56.514843 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.523783 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.525364 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.521922 2099 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:53:56.532409 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 383725us
I20250114 02:53:56.533756 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.536893 1661 raft_consensus.cc:2388] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:56.540386 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.545511 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.536010 1735 raft_consensus.cc:2388] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
W20250114 02:53:56.545646 1553 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.106.67:42283 timed out after 1.610s (SENT) [suppressed 7 similar messages]
I20250114 02:53:56.551337 2242 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:56.551875 2242 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.554210 1656 raft_consensus.cc:2463] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
W20250114 02:53:56.526203 2178 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.014s sys 0.039s
I20250114 02:53:56.532272 1653 raft_consensus.cc:2463] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:56.558881 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.563275 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.554275 2242 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.555277 2321 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:53:56.557631 2178 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.680s user 0.014s sys 0.039s
I20250114 02:53:56.533180 2099 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.554843 1657 raft_consensus.cc:2463] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:56.559769 2182 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.572243 2357 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.563023 2352 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.562747 2340 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.577936 1553 leader_election.cc:304] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:56.572309 2260 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.571894 2321 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.571218 2322 consensus_queue.cc:1035] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20250114 02:53:56.590886 2099 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.591445 2260 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:56.598693 2340 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.583930 2352 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.600732 2340 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:56.603394 1553 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:56.572489 1655 raft_consensus.cc:1235] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:56.569204 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.604270 1553 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:56.612597 1553 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:56.614135 1553 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:56.607560 1628 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:56.574426 2346 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.539166 2307 consensus_queue.cc:237] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.617102 1553 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:56.617892 1628 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:56.622171 1628 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:56.610320 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48636 (request call id 245) took 1125 ms. Trace:
I20250114 02:53:56.600732 2352 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.624496 1553 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:56.603252 2348 raft_consensus.cc:2798] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:56.601943 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.627107 1628 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:56.627789 1553 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:56.610769 2233 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:56.603852 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.576027 2242 consensus_queue.cc:1046] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20250114 02:53:56.629259 1628 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:56.613504 2248 raft_consensus.cc:2798] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:56.616614 2106 raft_consensus.cc:2798] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:56.580660 2182 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.622380 2303 raft_consensus.cc:2798] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:56.619046 2322 raft_consensus.cc:2798] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:56.623781 2302 raft_consensus.cc:2798] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:56.621953 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.618898 2360 raft_consensus.cc:2798] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:56.626228 2216 raft_consensus.cc:2798] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:56.627507 2346 raft_consensus.cc:2798] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:56.630741 2268 raft_consensus.cc:2798] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:56.623615 1658 rpcz_store.cc:276] 0114 02:53:55.485578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:55.543327 (+ 57749us) service_pool.cc:225] Handling call
0114 02:53:56.579958 (+1036631us) spinlock_profiling.cc:243] Waited 99.6 ms on lock 0x7b5400365954. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:56.602806 (+ 22848us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":229058432}
I20250114 02:53:56.619140 2099 consensus_queue.cc:1035] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20250114 02:53:56.632114 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.717967 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:53:56.725966 2374 log.cc:926] Time spent T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20250114 02:53:56.634110 2248 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:56.639374 1628 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:56.635509 2303 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:56.700896 1735 raft_consensus.cc:2388] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:56.633972 2106 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:56.774711 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 220) took 1232 ms. Trace:
I20250114 02:53:56.774689 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.638028 2302 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:56.638747 2268 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:56.638233 2216 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:56.633312 2233 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:56.775961 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 136697us
I20250114 02:53:56.776197 2284 raft_consensus.cc:3049] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:56.778192 1660 rpcz_store.cc:276] 0114 02:53:55.549909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:55.551974 (+ 2065us) service_pool.cc:225] Handling call
0114 02:53:55.552246 (+ 272us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:55.918460 (+366214us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:56.774475 (+856015us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b540034c954. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:53:56.774656 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":823975168}
I20250114 02:53:56.775468 2303 raft_consensus.cc:3054] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.636732 2322 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:56.631629 2246 raft_consensus.cc:2798] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:53:56.700747 2327 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:56.632963 1653 raft_consensus.cc:2463] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:56.777348 2248 raft_consensus.cc:3054] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:56.785471 2202 raft_consensus.cc:2798] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:56.632642 2373 raft_consensus.cc:2798] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:56.783203 2106 raft_consensus.cc:3054] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:53:56.786621 2378 log.cc:926] Time spent T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.137s user 0.005s sys 0.005s
I20250114 02:53:56.784289 2302 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:56.820294 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:56.820804 1699 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36436 dropped due to backpressure: service queue is full with 50 items
I20250114 02:53:56.785012 2268 raft_consensus.cc:3054] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:53:56.777604 1656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48636 (request call id 243) took 1352 ms (client timeout 1538 ms). Trace:
I20250114 02:53:56.633616 1733 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:56.809353 2202 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:56.786265 2216 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:56.808430 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:56.810159 1628 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:56.811703 1553 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
W20250114 02:53:56.871889 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 60270.1us
W20250114 02:53:56.872193 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 62163.5us
I20250114 02:53:56.828439 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:56.803968 2251 consensus_queue.cc:1046] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.064s
I20250114 02:53:56.792011 2322 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.787885 2284 raft_consensus.cc:738] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:56.873596 2251 raft_consensus.cc:2798] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:56.813887 1660 raft_consensus.cc:1235] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:56.873986 2251 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:56.874394 2251 raft_consensus.cc:3054] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.874761 2099 raft_consensus.cc:2798] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:56.875253 2099 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:56.875679 2099 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.875928 1628 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:56.877465 1628 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:56.878862 2365 raft_consensus.cc:2798] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:56.816191 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:56.813059 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:56.818881 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.819360 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:56.780478 2346 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:56.804952 2246 raft_consensus.cc:695] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:56.820915 1584 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:56.878896 2229 raft_consensus.cc:2798] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:56.791253 2327 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:56.809604 2373 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:56.879433 2365 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:56.885350 1628 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:56.829860 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:56.828899 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.827991 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:56.871006 2202 raft_consensus.cc:3054] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.822490 1728 raft_consensus.cc:2463] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:56.879189 2368 consensus_queue.cc:1035] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250114 02:53:56.931582 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 46311.5us
I20250114 02:53:56.932621 1628 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:56.934651 1660 raft_consensus.cc:1235] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:56.890122 2346 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.888587 2360 raft_consensus.cc:695] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:56.922300 2327 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:56.891435 1735 raft_consensus.cc:2463] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:56.889250 1582 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:56.891875 2229 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:56.945192 1628 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:56.920256 2365 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.922717 2373 raft_consensus.cc:3054] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
W20250114 02:53:56.826263 1656 rpcz_store.cc:269] 0114 02:53:55.421333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:55.422574 (+ 1241us) service_pool.cc:225] Handling call
0114 02:53:56.721405 (+1298831us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540021e3d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:56.773523 (+ 52118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2328672256}
I20250114 02:53:56.893108 2348 raft_consensus.cc:695] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:56.821214 1699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36436 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 337)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 349)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 342)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 345)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 248)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 352)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 348)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 251)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 354)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 338)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 346)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 351)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 350)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 347)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 355)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 344)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 339)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 343)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 335)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 215)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 220)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 222)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 224)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 236)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 237)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 239)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 240)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 241)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 255)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 340)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 341)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 353)
I20250114 02:53:56.968071 1628 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:56.948412 2334 raft_consensus.cc:2798] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:56.952796 2350 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.964150 2229 raft_consensus.cc:3054] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:56.925278 2248 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.965466 2242 raft_consensus.cc:3049] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:56.980690 2242 raft_consensus.cc:738] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:56.973505 2176 raft_consensus.cc:2798] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250114 02:53:56.983556 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 79680.3us
I20250114 02:53:56.986759 1628 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:56.972565 2114 raft_consensus.cc:2798] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:56.973186 2334 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:56.974738 2182 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:56.979513 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:56.981287 2372 log.cc:926] Time spent T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.209s user 0.000s sys 0.001s
I20250114 02:53:56.993314 1628 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:57.002640 1579 raft_consensus.cc:2388] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:57.003639 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.004261 1734 raft_consensus.cc:2463] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:56.976204 2336 consensus_queue.cc:1035] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20250114 02:53:56.983728 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.025259 2368 raft_consensus.cc:2798] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:56.997700 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.998879 2114 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:57.000686 2182 raft_consensus.cc:2798] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:56.992352 2176 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:56.999919 2233 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.030345 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 302) took 1061 ms. Trace:
I20250114 02:53:57.030453 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:56.972412 2246 consensus_queue.cc:237] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.981241 2379 consensus_queue.cc:1046] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20250114 02:53:56.999061 2334 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:57.043203 2307 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:57.045728 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.065680 2307 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.067736 2307 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:57.069721 1484 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39724 (request call id 110) took 3726 ms. Trace:
W20250114 02:53:57.046384 1553 leader_election.cc:336] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Timed out: RequestConsensusVote RPC to 127.0.106.67:42283 timed out after 1.511s (SENT)
I20250114 02:53:57.043179 2392 consensus_queue.cc:1046] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:53:57.060663 1586 rpcz_store.cc:276] 0114 02:53:55.969190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:55.970021 (+ 831us) service_pool.cc:225] Handling call
0114 02:53:57.030202 (+1060181us) spinlock_profiling.cc:243] Waited 80.8 ms on lock 0x7b54001ab3d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:57.030292 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":185930624}
I20250114 02:53:57.092592 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.093624 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.057271 2114 raft_consensus.cc:3054] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:57.006048 2262 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.029037 2368 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:57.086647 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.086620 2365 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:56.992611 2360 consensus_queue.cc:237] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.023667 2348 consensus_queue.cc:237] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.087949 2380 raft_consensus.cc:3049] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:57.089313 2216 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.089888 2251 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.090546 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.032377 2385 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:57.056658 2327 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.091902 1553 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
W20250114 02:53:57.118645 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:53:57.060309 2182 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:57.120699 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.140654 2182 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:57.141222 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:57.141484 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.142684 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.143270 1734 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:57.144030 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.144582 1734 raft_consensus.cc:2463] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:57.145385 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.146070 1734 raft_consensus.cc:2388] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 1: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:57.062179 2176 raft_consensus.cc:3054] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:57.146337 1653 raft_consensus.cc:1270] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:57.148411 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.122004 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.006134 2242 consensus_queue.cc:260] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:57.149784 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 122091us
I20250114 02:53:57.053475 2373 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.071909 1658 raft_consensus.cc:1270] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:57.071702 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 323) took 1051 ms. Trace:
I20250114 02:53:57.124217 2368 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
W20250114 02:53:57.132037 2381 log.cc:926] Time spent T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.126s user 0.001s sys 0.000s
I20250114 02:53:57.150518 1628 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
W20250114 02:53:57.152667 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 106306us
I20250114 02:53:57.154402 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:53:57.157279 1553 leader_election.cc:336] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Timed out: RequestConsensusVote RPC to 127.0.106.67:42283 timed out after 1.518s (SENT)
I20250114 02:53:57.157655 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.134243 2380 raft_consensus.cc:738] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:57.157842 1553 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:57.159749 2350 raft_consensus.cc:2743] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:57.161523 2202 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.162828 1657 raft_consensus.cc:2388] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:57.163731 2099 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.164994 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.181742 1585 raft_consensus.cc:2463] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:57.182680 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.183270 1585 raft_consensus.cc:2463] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:57.183089 2268 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.184340 1587 raft_consensus.cc:2388] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:57.184230 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.184806 1585 raft_consensus.cc:2463] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:53:57.185612 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:57.186105 1585 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:57.149818 1731 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:57.151929 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.188138 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:57.155485 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.166802 2251 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.191913 2176 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.193104 2202 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.193737 1660 raft_consensus.cc:1235] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.193802 2176 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.166893 2248 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.158279 1734 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:57.168205 1656 raft_consensus.cc:1235] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: []
I20250114 02:53:57.153273 1732 rpcz_store.cc:276] 0114 02:53:56.011650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:56.014866 (+ 3216us) service_pool.cc:225] Handling call
0114 02:53:56.294645 (+279779us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b5800023d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:53:56.294888 (+ 243us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:53:56.295334 (+ 446us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:56.295380 (+ 46us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:53:56.295961 (+ 581us) log.cc:844] Serialized 26 byte log entry
0114 02:53:56.564109 (+268148us) raft_consensus.cc:1640] Marking committed up to 0
0114 02:53:56.566175 (+ 2066us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:56.570875 (+ 4700us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:53:57.026100 (+455225us) raft_consensus.cc:1680] finished
0114 02:53:57.028069 (+ 1969us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:57.063455 (+ 35386us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6558,"raft.queue_time_us":15439,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":1738,"spinlock_wait_cycles":5134720,"thread_start_us":261923,"threads_started":1,"wal-append.queue_time_us":337292}
I20250114 02:53:57.169076 2388 raft_consensus.cc:2798] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:57.169780 2373 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.156718 2182 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.251372 2388 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:57.151695 2385 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.251914 2388 raft_consensus.cc:3054] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:57.176477 2365 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.176743 2321 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.186956 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.171207 2302 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.191038 2380 consensus_queue.cc:260] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.196060 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:57.256275 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.256760 1587 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:57.185233 1699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36436 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 338)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 346)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 351)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 350)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 347)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 355)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 369)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 344)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 339)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 343)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 335)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 215)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 220)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 222)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 224)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 236)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 237)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 239)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 240)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 241)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 255)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 340)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 341)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 353)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 359)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 361)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 362)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 363)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 364)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 365)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 366)
I20250114 02:53:57.154129 2392 raft_consensus.cc:2743] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:57.179127 2386 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:57.258955 2386 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:57.251421 2391 log.cc:926] Time spent T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20250114 02:53:57.263368 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.254627 1628 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:57.252830 2182 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.265686 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.266300 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.260330 2307 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:57.272148 1628 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:57.272295 1734 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:57.275682 1628 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:57.277436 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.273955 2385 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.278095 1734 raft_consensus.cc:2388] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:57.278415 2216 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.278950 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.279855 1734 raft_consensus.cc:2388] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:57.279804 2322 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.280674 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:57.281011 1653 raft_consensus.cc:1235] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250114 02:53:57.281827 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.282779 1734 raft_consensus.cc:2463] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:57.283694 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:57.284363 1734 raft_consensus.cc:3054] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:57.273602 2365 raft_consensus.cc:2798] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:57.290854 2365 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:57.291162 2302 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.291227 2365 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:57.276463 2350 raft_consensus.cc:2798] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:57.292155 2350 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:57.292483 2350 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:57.274446 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 221) took 1710 ms. Trace:
I20250114 02:53:57.273594 2380 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:57.303901 2386 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.304706 1734 raft_consensus.cc:2463] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:57.305791 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.306396 1734 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:57.294265 1654 rpcz_store.cc:276] 0114 02:53:55.577214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:55.584538 (+ 7324us) service_pool.cc:225] Handling call
0114 02:53:55.786990 (+202452us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:53:55.908091 (+121101us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.262001 (+1353910us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b5400436fa0. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce6065990 00007fbce60d8b42 00007fbce60d4f26 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782
0114 02:53:57.273251 (+ 11250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1518328192}
I20250114 02:53:57.307811 1654 raft_consensus.cc:1270] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:57.307710 2334 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.307886 2388 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.309191 1654 raft_consensus.cc:1270] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:57.309703 2334 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.309896 2388 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.310338 2327 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:57.302776 2303 consensus_meta.cc:287] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.506s user 0.007s sys 0.004s
I20250114 02:53:57.311148 2268 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.310880 2303 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.311565 1654 raft_consensus.cc:1270] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:57.316421 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.317224 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 330) took 1244 ms. Trace:
I20250114 02:53:57.317560 1736 rpcz_store.cc:276] 0114 02:53:56.073287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:56.158145 (+ 84858us) service_pool.cc:225] Handling call
0114 02:53:56.625869 (+467724us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b5800023d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315ed0e 0000000000353db9 00007fbceafbea50 00007fbceb084814 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:53:57.316344 (+690475us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b5800023d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315edac 00007fbce315ed16 0000000000353db9 00007fbceafbea50 00007fbceb084814 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:53:57.317157 (+ 813us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8919168}
I20250114 02:53:57.306440 2114 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.320605 2114 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.324049 1583 raft_consensus.cc:1235] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.324676 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 313) took 1310 ms. Trace:
I20250114 02:53:57.281790 2322 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.325656 2229 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.329703 2229 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.275826 2307 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.070066 1484 rpcz_store.cc:276] 0114 02:53:53.342822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:53.342969 (+ 147us) service_pool.cc:225] Handling call
0114 02:53:53.380293 (+ 37324us) spinlock_profiling.cc:243] Waited 880 us on lock 0x7b5400004738. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebdf9d93 00007fbcebef1dee 00007fbcea444fc4 00007fbcea444f52 00007fbcea444e81 00007fbcea444dfc 00007fbcea4440b2 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:57.069640 (+3689347us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:53:53.546238 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:53:53.632143 (+ 85905us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:53:53.632192 (+ 49us) write_op.cc:435] Acquired schema lock
0114 02:53:53.632211 (+ 19us) tablet.cc:659] Decoding operations
0114 02:53:53.632559 (+ 348us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:53:53.632645 (+ 86us) write_op.cc:641] Partition lock acquired for write op
0114 02:53:53.632669 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:53:53.632946 (+ 277us) tablet.cc:698] Row locks acquired
0114 02:53:53.632982 (+ 36us) write_op.cc:260] PREPARE: finished
0114 02:53:53.633121 (+ 139us) write_op.cc:270] Start()
0114 02:53:53.633235 (+ 114us) write_op.cc:276] Timestamp: P: 1736823233633082 usec, L: 0
0114 02:53:53.633268 (+ 33us) op_driver.cc:348] REPLICATION: starting
0114 02:53:53.714775 (+ 81507us) log.cc:844] Serialized 537 byte log entry
0114 02:53:54.339627 (+624852us) op_driver.cc:464] REPLICATION: finished
0114 02:53:54.631313 (+291686us) write_op.cc:301] APPLY: starting
0114 02:53:54.631408 (+ 95us) tablet.cc:1364] starting BulkCheckPresence
0114 02:53:54.631526 (+ 118us) tablet.cc:1367] finished BulkCheckPresence
0114 02:53:54.634247 (+ 2721us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:53:54.634622 (+ 375us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:53:54.634905 (+ 283us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:53:54.634987 (+ 82us) write_op.cc:312] APPLY: finished
0114 02:53:54.635295 (+ 308us) log.cc:844] Serialized 22 byte log entry
0114 02:53:54.635468 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:53:54.635552 (+ 84us) write_op.cc:454] Released schema lock
0114 02:53:54.635605 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0114 02:53:54.635903 (+ 298us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:53:54.635931 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":35012,"reported_tablets":10,"spinlock_wait_cycles":2024192,"child_traces":[["op",{"apply.queue_time_us":291365,"apply.run_cpu_time_us":2142,"apply.run_wall_time_us":4621,"num_ops":1,"prepare.queue_time_us":42450,"prepare.run_cpu_time_us":24783,"prepare.run_wall_time_us":282488,"raft.queue_time_us":423736,"raft.run_cpu_time_us":18018,"raft.run_wall_time_us":186179,"replication_time_us":706218,"thread_start_us":600401,"threads_started":4,"wal-append.queue_time_us":187887,"wal-append.run_cpu_time_us":86846,"wal-append.run_wall_time_us":1765313}]]}
I20250114 02:53:57.452301 2303 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.367544 2368 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.457239 2368 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.457204 2350 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.458345 2307 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:53:57.459162 2396 log.cc:926] Time spent T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20250114 02:53:57.459304 2350 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.324994 1583 rpcz_store.cc:276] 0114 02:53:56.014299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:56.247071 (+232772us) service_pool.cc:225] Handling call
0114 02:53:57.323690 (+1076619us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:53:57.323878 (+ 188us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.324443 (+ 565us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.324620 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49763072}
W20250114 02:53:57.364612 2106 consensus_meta.cc:287] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.540s user 0.007s sys 0.002s
I20250114 02:53:57.363899 2346 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.462437 2106 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.463780 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.464074 2346 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.464442 1727 raft_consensus.cc:2463] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:57.464668 2106 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.465574 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.465862 2365 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.467588 2365 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.469178 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:57.470899 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:57.471372 1727 raft_consensus.cc:3054] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:57.477859 1727 raft_consensus.cc:2463] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:57.478830 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.479342 1727 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:57.479947 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.480369 2284 consensus_queue.cc:260] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.480710 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.481117 2284 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:57.481199 1727 raft_consensus.cc:2463] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:57.485533 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 335) took 1345 ms. Trace:
I20250114 02:53:57.485807 1727 rpcz_store.cc:276] 0114 02:53:56.139690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.481710 (+1342020us) service_pool.cc:225] Handling call
0114 02:53:57.481974 (+ 264us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:53:57.482239 (+ 265us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:57.482268 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:53:57.482647 (+ 379us) log.cc:844] Serialized 26 byte log entry
0114 02:53:57.483551 (+ 904us) raft_consensus.cc:1640] Marking committed up to 0
0114 02:53:57.483592 (+ 41us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.483656 (+ 64us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:53:57.485369 (+ 1713us) raft_consensus.cc:1680] finished
0114 02:53:57.485384 (+ 15us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:57.485498 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":89,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"thread_start_us":591,"threads_started":1,"wal-append.queue_time_us":915}
I20250114 02:53:57.487588 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 212) took 1202 ms. Trace:
I20250114 02:53:57.487838 1727 rpcz_store.cc:276] 0114 02:53:56.284987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.487024 (+1202037us) service_pool.cc:225] Handling call
0114 02:53:57.487255 (+ 231us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.487377 (+ 122us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:57.487393 (+ 16us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:57.487418 (+ 25us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:57.487438 (+ 20us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.487470 (+ 32us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:57.487555 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.489905 1727 raft_consensus.cc:1235] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.490334 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 213) took 1205 ms. Trace:
I20250114 02:53:57.490569 1727 rpcz_store.cc:276] 0114 02:53:56.285261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.489540 (+1204279us) service_pool.cc:225] Handling call
0114 02:53:57.489782 (+ 242us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.490171 (+ 389us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.490302 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.490864 1586 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:57.492164 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 214) took 1206 ms. Trace:
I20250114 02:53:57.492520 1727 rpcz_store.cc:276] 0114 02:53:56.285566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.491384 (+1205818us) service_pool.cc:225] Handling call
0114 02:53:57.491675 (+ 291us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.491844 (+ 169us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:57.491871 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:57.491915 (+ 44us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:57.491964 (+ 49us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.492021 (+ 57us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:57.492119 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.493928 1727 raft_consensus.cc:1235] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.494516 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 215) took 1208 ms. Trace:
I20250114 02:53:57.494629 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.494961 1727 rpcz_store.cc:276] 0114 02:53:56.285837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.493470 (+1207633us) service_pool.cc:225] Handling call
0114 02:53:57.493755 (+ 285us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.494304 (+ 549us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.494470 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.495944 1731 raft_consensus.cc:1235] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.496062 1727 raft_consensus.cc:1235] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.496536 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 216) took 1210 ms. Trace:
I20250114 02:53:57.496592 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 219) took 1209 ms. Trace:
I20250114 02:53:57.496951 1731 rpcz_store.cc:276] 0114 02:53:56.286130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.495523 (+1209393us) service_pool.cc:225] Handling call
0114 02:53:57.495775 (+ 252us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.496316 (+ 541us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.496487 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.497004 1727 rpcz_store.cc:276] 0114 02:53:56.286947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.495663 (+1208716us) service_pool.cc:225] Handling call
0114 02:53:57.495897 (+ 234us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.496397 (+ 500us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.496548 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.498431 1731 raft_consensus.cc:1235] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.498621 1727 raft_consensus.cc:1235] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.499101 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 220) took 1211 ms. Trace:
I20250114 02:53:57.499302 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 221) took 1211 ms. Trace:
I20250114 02:53:57.499514 1731 rpcz_store.cc:276] 0114 02:53:56.287273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.498012 (+1210739us) service_pool.cc:225] Handling call
0114 02:53:57.498271 (+ 259us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.498883 (+ 612us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.499055 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.499697 1727 rpcz_store.cc:276] 0114 02:53:56.287579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.498176 (+1210597us) service_pool.cc:225] Handling call
0114 02:53:57.498435 (+ 259us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.499095 (+ 660us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.499259 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.500932 1731 raft_consensus.cc:1235] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.501380 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 223) took 1204 ms. Trace:
I20250114 02:53:57.501544 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 222) took 1204 ms. Trace:
I20250114 02:53:57.501950 1731 rpcz_store.cc:276] 0114 02:53:56.297118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.500504 (+1203386us) service_pool.cc:225] Handling call
0114 02:53:57.500769 (+ 265us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.501338 (+ 569us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.501497 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.501821 1727 rpcz_store.cc:276] 0114 02:53:56.297388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.500636 (+1203248us) service_pool.cc:225] Handling call
0114 02:53:57.500899 (+ 263us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.501067 (+ 168us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:57.501098 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:57.501143 (+ 45us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:57.501178 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.501240 (+ 62us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:57.501336 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.503350 1731 raft_consensus.cc:1235] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.503968 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 224) took 1204 ms. Trace:
I20250114 02:53:57.503973 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 225) took 1204 ms. Trace:
I20250114 02:53:57.504467 1731 rpcz_store.cc:276] 0114 02:53:56.299258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.502917 (+1203659us) service_pool.cc:225] Handling call
0114 02:53:57.503181 (+ 264us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.503747 (+ 566us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.503920 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.504554 1727 rpcz_store.cc:276] 0114 02:53:56.299499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.503120 (+1203621us) service_pool.cc:225] Handling call
0114 02:53:57.503403 (+ 283us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.503755 (+ 352us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.503920 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.505916 1731 raft_consensus.cc:1235] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.506007 1727 raft_consensus.cc:1235] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: []
I20250114 02:53:57.506609 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 226) took 1206 ms. Trace:
I20250114 02:53:57.506657 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 227) took 1206 ms. Trace:
I20250114 02:53:57.507076 1731 rpcz_store.cc:276] 0114 02:53:56.299719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.505479 (+1205760us) service_pool.cc:225] Handling call
0114 02:53:57.505749 (+ 270us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.506359 (+ 610us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.506532 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.507148 1727 rpcz_store.cc:276] 0114 02:53:56.299925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.505575 (+1205650us) service_pool.cc:225] Handling call
0114 02:53:57.505822 (+ 247us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.506431 (+ 609us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.506614 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.508832 1727 raft_consensus.cc:1270] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:57.509554 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 236) took 1136 ms. Trace:
I20250114 02:53:57.509096 1731 raft_consensus.cc:1270] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:57.510331 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 235) took 1164 ms. Trace:
I20250114 02:53:57.510048 1727 rpcz_store.cc:276] 0114 02:53:56.373147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.508160 (+1135013us) service_pool.cc:225] Handling call
0114 02:53:57.508518 (+ 358us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.509270 (+ 752us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.509489 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.510846 1731 rpcz_store.cc:276] 0114 02:53:56.345685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.508084 (+1162399us) service_pool.cc:225] Handling call
0114 02:53:57.508789 (+ 705us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.510089 (+ 1300us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.510287 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.511930 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 237) took 1065 ms. Trace:
I20250114 02:53:57.512207 1731 raft_consensus.cc:1270] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:57.512338 1727 rpcz_store.cc:276] 0114 02:53:56.446885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.511090 (+1064205us) service_pool.cc:225] Handling call
0114 02:53:57.511385 (+ 295us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.511727 (+ 342us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.511884 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.512876 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 239) took 1019 ms. Trace:
I20250114 02:53:57.513245 1731 rpcz_store.cc:276] 0114 02:53:56.493112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.511660 (+1018548us) service_pool.cc:225] Handling call
0114 02:53:57.511907 (+ 247us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.512657 (+ 750us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.512833 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.513789 1727 raft_consensus.cc:1270] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:57.514437 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 240) took 1007 ms. Trace:
I20250114 02:53:57.514458 1731 raft_consensus.cc:1235] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.514909 1727 rpcz_store.cc:276] 0114 02:53:56.508223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.513218 (+1004995us) service_pool.cc:225] Handling call
0114 02:53:57.513505 (+ 287us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.514229 (+ 724us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.514392 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.515823 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 241) took 1006 ms. Trace:
I20250114 02:53:57.516324 1731 rpcz_store.cc:276] 0114 02:53:56.515365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.514003 (+998638us) service_pool.cc:225] Handling call
0114 02:53:57.514306 (+ 303us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.515410 (+ 1104us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.515776 (+ 366us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:57.517020 1727 raft_consensus.cc:1270] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:57.517596 1731 raft_consensus.cc:1235] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.518015 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:57.519994 1579 raft_consensus.cc:3054] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 2
I20250114 02:53:57.522766 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.523430 1735 raft_consensus.cc:2463] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:57.534783 1485 rpcz_store.cc:275] Call kudu.master.MasterService.IsCreateTableDone from 127.0.0.1:39758 (request call id 38) took 1635 ms. Trace:
I20250114 02:53:57.535342 1485 rpcz_store.cc:276] 0114 02:53:55.536268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:55.546726 (+ 10458us) service_pool.cc:225] Handling call
0114 02:53:55.553685 (+ 6959us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:53:57.117957 (+1564272us) catalog_manager.cc:2246] Verify if the table creation is in progress for test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]
0114 02:53:57.118528 (+ 571us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1564052}
W20250114 02:53:57.540319 2395 log.cc:926] Time spent T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.276s user 0.001s sys 0.000s
I20250114 02:53:57.545655 1579 raft_consensus.cc:2463] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
W20250114 02:53:57.550139 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 274516us
I20250114 02:53:57.550585 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:57.551694 2387 raft_consensus.cc:2798] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:57.555874 2387 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:57.556217 2387 raft_consensus.cc:3054] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:57.552528 1628 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:57.559015 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:57.560547 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.556459 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:57.570480 2376 log.cc:926] Time spent T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.797s user 0.002s sys 0.000s
I20250114 02:53:57.573293 1655 raft_consensus.cc:1235] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:57.577967 2322 raft_consensus.cc:2798] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:57.578482 2322 raft_consensus.cc:695] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:57.576431 2387 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.580708 1702 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:57.582512 1702 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:57.590803 2326 raft_consensus.cc:2743] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:57.591450 2326 raft_consensus.cc:2743] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:57.592000 2319 consensus_queue.cc:1035] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:57.595317 1579 raft_consensus.cc:2388] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:57.595956 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.596455 1727 raft_consensus.cc:2463] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:53:57.587612 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.597128 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.597744 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:57.598165 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:57.598702 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.579661 2322 consensus_queue.cc:237] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.602552 2387 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.599819 1628 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:57.578461 2307 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:57.615197 2307 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.618111 2307 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:57.621610 2387 raft_consensus.cc:2798] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:57.622061 2387 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:57.622466 2387 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:57.621393 2357 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:57.623862 2357 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.625288 1628 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:57.626725 1628 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:57.628456 1628 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:57.630069 1628 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:57.631383 1628 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:57.648329 1628 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:57.651976 1628 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:57.674959 1628 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
W20250114 02:53:57.737397 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 62525.7us
W20250114 02:53:57.728418 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 146007us
I20250114 02:53:57.818732 1702 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:57.821846 1702 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:57.825922 2326 raft_consensus.cc:2743] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:57.673599 2176 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:57.824785 2368 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:57.645895 2114 raft_consensus.cc:2743] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:57.826920 2176 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:57.734436 2350 raft_consensus.cc:2798] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:57.673609 2229 raft_consensus.cc:2798] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:57.673993 2365 raft_consensus.cc:2743] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:57.854705 2229 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:57.861367 2229 raft_consensus.cc:3054] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:57.834828 2176 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:57.827492 2368 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:57.864019 2368 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:57.835927 2357 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.867192 2379 raft_consensus.cc:3049] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:57.867588 2379 raft_consensus.cc:738] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:57.868326 2379 consensus_queue.cc:260] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.869450 2379 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:57.673848 2339 raft_consensus.cc:2798] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:57.688194 2346 raft_consensus.cc:2798] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:57.848714 1702 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:57.825060 2307 raft_consensus.cc:2758] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250114 02:53:57.682127 1727 raft_consensus.cc:2371] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2. Re-sending same reply.
I20250114 02:53:57.604596 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 266)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 364)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 276)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 269)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 270)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 272)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 274)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 275)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 278)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 280)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 281)
I20250114 02:53:57.872473 2339 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:57.883447 2339 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:57.883708 2176 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.885762 2176 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.862423 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.872941 2346 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:57.874039 2350 raft_consensus.cc:695] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
W20250114 02:53:57.895615 1550 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items [suppressed 2 similar messages]
I20250114 02:53:57.892851 2346 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:57.894613 2307 raft_consensus.cc:2743] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:57.898478 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.910812 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.911872 1727 raft_consensus.cc:2388] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:57.912890 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.913465 2197 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:57.914007 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.913947 2197 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.915575 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.916337 1735 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:57.916764 2197 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.917346 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.919006 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:57.925614 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.925880 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.928611 2339 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:57.929298 1729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 325) took 1916 ms (client timeout 1656 ms). Trace:
W20250114 02:53:57.929703 1729 rpcz_store.cc:269] 0114 02:53:56.012519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:56.014737 (+ 2218us) service_pool.cc:225] Handling call
0114 02:53:57.925722 (+1910985us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b5800023d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:53:57.929213 (+ 3491us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":257803648}
I20250114 02:53:57.930797 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.907433 2350 consensus_queue.cc:237] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.903687 1733 raft_consensus.cc:2388] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:57.918432 1735 raft_consensus.cc:2405] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:53:57.895998 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 364)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 276)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 269)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 270)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 272)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 274)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 275)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 278)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 280)
I20250114 02:53:57.940160 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 364)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 276)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 269)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 270)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 272)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 274)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 275)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 278)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 280)
W20250114 02:53:57.938709 2352 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.794s user 0.000s sys 0.036s
I20250114 02:53:57.954501 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 364)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 276)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 269)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 270)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 272)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 274)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 275)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 278)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 280)
I20250114 02:53:57.944651 2251 raft_consensus.cc:2743] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:57.949090 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:57.965299 2357 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
W20250114 02:53:57.945796 1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 250) took 1404 ms (client timeout 1850 ms). Trace:
I20250114 02:53:57.944780 1736 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:57.971462 2387 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.980983 2387 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.981881 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:53:57.976640 1733 rpcz_store.cc:269] 0114 02:53:56.540206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.862111 (+1321905us) service_pool.cc:225] Handling call
0114 02:53:57.944702 (+ 82591us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 02:53:57.973448 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 816222us
W20250114 02:53:57.954953 2352 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.811s user 0.000s sys 0.036s
I20250114 02:53:57.973598 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:57.977527 2357 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.973506 2339 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:57.977437 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 276)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 269)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 270)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 272)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 274)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 275)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 278)
I20250114 02:53:58.002286 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 276)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 269)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 270)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 272)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 274)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 275)
I20250114 02:53:57.986032 1729 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:57.986467 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 353) took 1159 ms. Trace:
I20250114 02:53:57.974306 2368 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:57.982547 1728 raft_consensus.cc:2371] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1. Re-sending same reply.
I20250114 02:53:58.040016 1727 raft_consensus.cc:1270] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:58.040023 1735 raft_consensus.cc:1235] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.020817 1583 raft_consensus.cc:2388] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:58.022526 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 276)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 269)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 270)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 272)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 274)
W20250114 02:53:58.084420 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 235816us
I20250114 02:53:58.085433 1702 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:58.037719 1729 raft_consensus.cc:1270] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:58.108343 1702 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:58.131282 2319 raft_consensus.cc:2743] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:58.132259 2368 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:58.135504 2413 log.cc:926] Time spent T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250114 02:53:58.137554 2357 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.121537 2412 raft_consensus.cc:2798] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:58.140125 2412 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:58.140520 2412 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.142976 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 364) took 1009 ms. Trace:
I20250114 02:53:58.143457 1733 rpcz_store.cc:276] 0114 02:53:57.133146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.984092 (+850946us) service_pool.cc:225] Handling call
0114 02:53:57.984421 (+ 329us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:53:57.984764 (+ 343us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:57.984797 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:53:57.985245 (+ 448us) log.cc:844] Serialized 26 byte log entry
0114 02:53:58.011086 (+ 25841us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:58.019422 (+ 8336us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:58.027412 (+ 7990us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:53:58.142712 (+115300us) raft_consensus.cc:1680] finished
0114 02:53:58.142749 (+ 37us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:58.142909 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8928,"raft.queue_time_us":1109,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":12087,"spinlock_wait_cycles":1565952,"thread_start_us":22517,"threads_started":1,"wal-append.queue_time_us":147765}
I20250114 02:53:58.035949 2229 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.123910 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 276)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 269)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 270)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 272)
W20250114 02:53:58.158771 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 171432us
W20250114 02:53:58.043406 1728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 249) took 1511 ms (client timeout 1781 ms). Trace:
I20250114 02:53:58.152832 2352 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:58.015257 2346 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.035682 1731 rpcz_store.cc:276] 0114 02:53:56.817560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.518330 (+700770us) service_pool.cc:225] Handling call
0114 02:53:57.518692 (+ 362us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.520081 (+ 1389us) raft_consensus.cc:1529] Early marking committed up to index 1
0114 02:53:57.728474 (+208393us) log.cc:844] Serialized 14 byte log entry
0114 02:53:57.945958 (+217484us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:53:57.947239 (+ 1281us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:57.972912 (+ 25673us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:57.974531 (+ 1619us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:57.977169 (+ 2638us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26821,"prepare.queue_time_us":4247,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":2970,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":7278,"thread_start_us":193364,"threads_started":2,"wal-append.queue_time_us":125066}
I20250114 02:53:58.163071 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.167174 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.167968 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.168329 2251 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:58.168982 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:58.169699 1729 raft_consensus.cc:2388] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:58.169812 1553 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.168787 2251 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.170611 2323 raft_consensus.cc:2743] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:58.170317 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:58.172538 2229 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.172495 1553 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.172713 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.173460 1729 raft_consensus.cc:2388] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:58.173300 2323 raft_consensus.cc:2743] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:58.174439 2251 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.173267 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.175316 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.176046 1729 raft_consensus.cc:2388] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 1: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:58.178042 2412 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.178402 2346 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.165618 2352 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.180043 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:58.180454 2352 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.163178 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.162261 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 269)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 270)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 271)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 272)
I20250114 02:53:58.175313 1553 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.184823 1553 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:58.180795 2412 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:58.185824 1553 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.182030 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
W20250114 02:53:58.164144 1728 rpcz_store.cc:269] 0114 02:53:56.532179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.981534 (+1449355us) service_pool.cc:225] Handling call
0114 02:53:58.042871 (+ 61337us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:58.184655 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 269)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 270)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 271)
I20250114 02:53:58.187244 1553 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.185782 2323 raft_consensus.cc:2743] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:58.186825 2284 raft_consensus.cc:3054] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:53:58.188509 2379 raft_consensus.cc:2743] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:58.188839 1553 leader_election.cc:304] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:58.189539 2379 raft_consensus.cc:2798] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:58.189901 2379 raft_consensus.cc:491] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:58.190251 2379 raft_consensus.cc:3054] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.192811 2414 consensus_queue.cc:1046] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.193914 2106 raft_consensus.cc:3049] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:58.194257 2106 raft_consensus.cc:738] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:58.186064 2251 raft_consensus.cc:2798] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:58.195645 2348 consensus_queue.cc:1046] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.187561 1733 raft_consensus.cc:3054] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.200381 2348 raft_consensus.cc:3049] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:58.200649 2385 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:58.200716 2348 raft_consensus.cc:738] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:58.201172 2385 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.201550 2348 consensus_queue.cc:260] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.202800 2348 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.195540 2106 consensus_queue.cc:260] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.196593 2380 consensus_queue.cc:1046] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.203663 2106 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.203953 2380 raft_consensus.cc:3049] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:58.204291 2380 raft_consensus.cc:738] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:58.204970 2380 consensus_queue.cc:260] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.206111 2380 raft_consensus.cc:3054] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.206970 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.207583 1661 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:58.208014 1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48632 (request call id 261) took 1726 ms. Trace:
I20250114 02:53:58.210129 2385 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.208305 1661 rpcz_store.cc:276] 0114 02:53:56.481217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:56.565362 (+ 84145us) service_pool.cc:225] Handling call
0114 02:53:58.206830 (+1641468us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b5800002d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:53:58.207962 (+ 1132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2694400}
I20250114 02:53:58.198292 2303 consensus_queue.cc:1046] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.198613 2246 consensus_queue.cc:1046] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.213760 2246 raft_consensus.cc:3049] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:58.213807 2303 raft_consensus.cc:3049] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:58.214226 2246 raft_consensus.cc:738] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:58.214318 2303 raft_consensus.cc:738] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:57.270418 1587 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:58.215171 2246 consensus_queue.cc:260] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.215204 2303 consensus_queue.cc:260] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.216396 2246 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.216501 2303 raft_consensus.cc:3054] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.198923 2268 consensus_queue.cc:1035] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.213825 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.219457 1661 raft_consensus.cc:2388] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 2: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:58.219733 2268 consensus_queue.cc:1035] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.219913 2302 consensus_queue.cc:1035] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:58.220279 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.220753 1661 raft_consensus.cc:3054] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.221032 2268 consensus_queue.cc:1035] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.221150 2302 consensus_queue.cc:1035] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.212328 1658 raft_consensus.cc:1235] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.223258 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 269) took 1593 ms. Trace:
I20250114 02:53:58.223599 1658 rpcz_store.cc:276] 0114 02:53:56.629970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.311931 (+681961us) service_pool.cc:225] Handling call
0114 02:53:58.212132 (+900201us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:58.223014 (+ 10882us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:58.223204 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:58.224144 1553 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.224221 2360 consensus_queue.cc:1046] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:58.224756 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.224938 2302 consensus_queue.cc:1035] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.225538 2386 raft_consensus.cc:3049] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:58.225874 2386 raft_consensus.cc:738] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:58.225944 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.195039 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 269)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 270)
I20250114 02:53:58.226665 2386 consensus_queue.cc:260] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.215863 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 336) took 1389 ms. Trace:
I20250114 02:53:58.227764 2386 raft_consensus.cc:3054] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.227564 1587 rpcz_store.cc:276] 0114 02:53:56.826974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.186475 (+359501us) service_pool.cc:225] Handling call
0114 02:53:58.215812 (+1029337us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:58.228674 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:58.229709 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:58.230019 2368 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:58.230510 2368 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.231580 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.237135 1658 raft_consensus.cc:3054] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.232548 2368 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.235823 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 269)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 270)
I20250114 02:53:58.238235 1661 raft_consensus.cc:2463] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:58.239269 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.239972 1661 raft_consensus.cc:2388] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 2: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:58.241012 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.241583 1661 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.242956 2352 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:58.243427 2352 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.244382 1658 raft_consensus.cc:2463] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:53:58.245360 2352 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.245837 2352 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:58.245612 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.246675 1658 raft_consensus.cc:2388] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:58.247470 2346 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:58.247639 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.247960 2346 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.248925 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.249696 1658 raft_consensus.cc:2388] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 1: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:58.249809 2346 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.250279 2346 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:58.234169 1553 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.236474 1653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 248) took 1787 ms. Trace:
I20250114 02:53:58.254724 1553 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:58.232101 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:58.255786 1553 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.250602 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.225137 2360 raft_consensus.cc:2743] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:58.239130 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 269)
I20250114 02:53:58.253201 1662 raft_consensus.cc:1235] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: []
I20250114 02:53:58.257563 1661 raft_consensus.cc:2463] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:53:58.259719 1553 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.199625 2251 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:58.232622 2099 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.246335 2352 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.255390 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 332) took 1459 ms. Trace:
I20250114 02:53:58.254127 2346 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.258973 2323 raft_consensus.cc:2798] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:58.259706 2360 raft_consensus.cc:2743] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:58.260102 1587 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.274699 2412 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:58.275249 2412 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.275779 2346 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.262392 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.277012 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.277601 2412 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:58.277928 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.265131 1553 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.278944 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.279465 1661 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:58.280151 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.280567 1553 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.281028 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.260701 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 238) took 1901 ms. Trace:
I20250114 02:53:58.266332 2251 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.265743 2348 raft_consensus.cc:2743] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:58.254844 1653 rpcz_store.cc:276] 0114 02:53:56.448718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.147128 (+698410us) service_pool.cc:225] Handling call
0114 02:53:57.147449 (+ 321us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:53:57.281444 (+133995us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:58.236237 (+954793us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b540021f2d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:53:58.236419 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":306598784}
I20250114 02:53:58.254717 2385 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.194425 2284 raft_consensus.cc:2743] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:53:58.274578 2323 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:58.281332 2246 raft_consensus.cc:2743] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:53:58.281747 2360 raft_consensus.cc:2743] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:58.282471 1553 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:58.272946 1581 rpcz_store.cc:276] 0114 02:53:56.796078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.149300 (+353222us) service_pool.cc:225] Handling call
0114 02:53:58.255331 (+1106031us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:58.284031 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 245) took 1921 ms. Trace:
I20250114 02:53:58.290526 1587 raft_consensus.cc:2463] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:58.293334 1628 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:58.293605 1661 raft_consensus.cc:2388] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:58.261832 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
I20250114 02:53:58.293879 1662 rpcz_store.cc:276] 0114 02:53:56.359596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:58.238549 (+1878953us) service_pool.cc:225] Handling call
0114 02:53:58.253008 (+ 14459us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:58.260462 (+ 7454us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:58.260650 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:58.299685 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.300050 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.300347 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:58.321504 1653 raft_consensus.cc:2388] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:58.321794 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:58.322513 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.300431 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.323952 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.324177 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.322186 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 264)
I20250114 02:53:58.302469 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.325147 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:58.326095 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.301446 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.327420 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:58.326146 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
I20250114 02:53:58.328385 1727 raft_consensus.cc:2463] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:58.329165 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:58.330150 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:58.307278 1728 raft_consensus.cc:1235] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:53:58.305145 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48632 (request call id 263) took 1764 ms. Trace:
I20250114 02:53:58.312773 2348 raft_consensus.cc:2798] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:58.311072 2323 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.313452 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:58.314260 2352 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.318696 1628 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.319375 1702 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:58.334404 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.312464 1553 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.325309 1729 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.306984 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.324898 1653 raft_consensus.cc:2388] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:58.335323 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.335973 1628 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.310330 1733 raft_consensus.cc:2463] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:53:58.338580 1553 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.320415 2346 raft_consensus.cc:2798] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:58.331044 1736 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.341361 1628 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:58.342957 1553 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.343827 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:58.344414 1553 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.339745 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 263)
I20250114 02:53:58.345642 1628 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
W20250114 02:53:58.356726 1585 consensus_meta.cc:287] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 1.170s user 0.014s sys 0.003s
I20250114 02:53:58.333740 2348 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:58.362771 1585 raft_consensus.cc:2463] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:58.364961 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 353) took 1185 ms. Trace:
I20250114 02:53:58.365247 1585 rpcz_store.cc:276] 0114 02:53:57.179783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.185401 (+ 5618us) service_pool.cc:225] Handling call
0114 02:53:58.364911 (+1179510us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:58.366351 1581 raft_consensus.cc:2388] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:58.367069 2385 raft_consensus.cc:2743] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:58.357084 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:58.367816 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.368276 1728 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:58.369235 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.337639 2099 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:58.369757 1658 raft_consensus.cc:2463] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:58.369940 2099 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:58.370296 2099 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.370849 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.371665 1658 raft_consensus.cc:2388] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:58.372437 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.372568 1731 raft_consensus.cc:2388] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:58.373449 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.373961 1731 raft_consensus.cc:3054] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.366755 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.375722 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.376765 1662 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:53:58.377506 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.378021 1662 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:53:58.339135 2284 raft_consensus.cc:2743] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:53:58.378654 2303 consensus_queue.cc:1035] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20250114 02:53:58.380774 2348 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.368469 1661 raft_consensus.cc:2388] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:58.332813 1657 rpcz_store.cc:276] 0114 02:53:56.541056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:56.602446 (+ 61390us) service_pool.cc:225] Handling call
0114 02:53:58.304959 (+1702513us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b5400365454. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:58.305090 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1184517504}
I20250114 02:53:58.383180 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.383570 1657 raft_consensus.cc:1235] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.383837 1661 raft_consensus.cc:2388] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:58.384136 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 273) took 1339 ms. Trace:
I20250114 02:53:58.384495 1657 rpcz_store.cc:276] 0114 02:53:57.045231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:58.383123 (+1337892us) service_pool.cc:225] Handling call
0114 02:53:58.383411 (+ 288us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:58.383928 (+ 517us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:58.384087 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:58.385515 1661 raft_consensus.cc:1235] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.386137 1657 raft_consensus.cc:1235] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.358219 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.362310 2248 raft_consensus.cc:2798] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:58.387105 1733 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.387315 2248 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:58.387701 2248 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.317137 1655 rpcz_store.cc:276] 0114 02:53:56.362161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.142248 (+780087us) service_pool.cc:225] Handling call
0114 02:53:57.142534 (+ 286us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.577339 (+434805us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:58.283731 (+706392us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b54000c1a54. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:53:58.283957 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":990260992}
I20250114 02:53:58.389353 1655 raft_consensus.cc:1235] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.369771 1728 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.357576 2379 raft_consensus.cc:513] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.363574 2346 raft_consensus.cc:695] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:58.391803 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:58.391906 2346 consensus_queue.cc:237] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.392388 2379 leader_election.cc:290] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.393200 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48636 (request call id 255) took 2462 ms. Trace:
I20250114 02:53:58.361976 2380 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:58.357112 2251 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.393720 2380 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:58.394117 2380 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.393617 1659 rpcz_store.cc:276] 0114 02:53:55.942351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:55.945618 (+ 3267us) service_pool.cc:225] Handling call
0114 02:53:55.963858 (+ 18240us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:56.508923 (+545065us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:58.392984 (+1884061us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b54001243d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:53:58.393145 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1214543744}
I20250114 02:53:58.395133 2251 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.390383 1655 raft_consensus.cc:1235] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3. Ops: []
I20250114 02:53:58.347712 2363 raft_consensus.cc:2743] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:53:58.370908 2106 raft_consensus.cc:2743] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:58.347985 2368 raft_consensus.cc:2758] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250114 02:53:58.398870 1658 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:53:58.335131 1727 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:58.382550 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 241) took 2021 ms. Trace:
W20250114 02:53:58.400905 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 51748us
I20250114 02:53:58.401120 1660 rpcz_store.cc:276] 0114 02:53:56.360830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:56.947002 (+586172us) service_pool.cc:225] Handling call
0114 02:53:56.965398 (+ 18396us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.194179 (+228781us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:58.382362 (+1188183us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b5400365954. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:53:58.382504 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":524792576}
I20250114 02:53:58.386538 1661 raft_consensus.cc:1235] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.347406 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
I20250114 02:53:58.346014 1735 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:58.404400 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:58.404881 1735 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:58.406141 1731 raft_consensus.cc:2463] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:58.406912 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.342289 2352 raft_consensus.cc:2743] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:58.407024 1661 raft_consensus.cc:1235] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3. Ops: []
W20250114 02:53:58.408843 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 63253.5us
I20250114 02:53:58.411306 1553 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.413832 1661 raft_consensus.cc:1235] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250114 02:53:58.414788 1660 raft_consensus.cc:1235] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3. Ops: []
I20250114 02:53:58.399976 1658 raft_consensus.cc:1235] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.415570 2303 consensus_queue.cc:1035] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.416301 1660 raft_consensus.cc:1235] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3. Ops: []
I20250114 02:53:58.416814 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:58.416977 2386 consensus_queue.cc:1035] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.417497 1653 raft_consensus.cc:1235] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.400170 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.407464 2368 raft_consensus.cc:2798] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:58.433257 2368 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:58.433341 2246 consensus_queue.cc:1035] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.433720 2368 raft_consensus.cc:3054] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.432358 2262 consensus_queue.cc:1035] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250114 02:53:58.434917 1553 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.412910 2379 raft_consensus.cc:2743] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:53:58.435806 2379 raft_consensus.cc:2743] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:58.436271 1553 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.437281 2386 raft_consensus.cc:2743] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:58.419718 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.437278 1553 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.407863 2251 consensus_queue.cc:1035] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20250114 02:53:58.420671 1657 raft_consensus.cc:1235] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3. Ops: []
I20250114 02:53:58.422930 2099 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.423933 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.425088 2360 consensus_queue.cc:1035] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:53:58.426595 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.442049 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 371) took 1257 ms. Trace:
I20250114 02:53:58.428517 2229 consensus_queue.cc:1035] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.429404 1628 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.419734 1661 raft_consensus.cc:1235] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.446851 1661 raft_consensus.cc:1235] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.443827 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.448062 1735 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.449272 2387 raft_consensus.cc:2798] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:58.449476 2262 raft_consensus.cc:3054] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.450793 2387 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:58.451136 2387 raft_consensus.cc:3054] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.442440 1734 rpcz_store.cc:276] 0114 02:53:57.184597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.333837 (+149240us) service_pool.cc:225] Handling call
0114 02:53:58.441982 (+1108145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:58.449438 2357 consensus_queue.cc:1046] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.006s
I20250114 02:53:58.455389 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.456656 1553 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
W20250114 02:53:58.448702 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 113872us
I20250114 02:53:58.439093 1659 raft_consensus.cc:1235] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.462965 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.462977 1553 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:58.460145 1702 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:58.460634 1628 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.465837 1553 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:58.467072 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.468606 1734 raft_consensus.cc:1270] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:58.467383 1553 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:58.466589 1628 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.466396 1702 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:58.471779 1702 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:58.467952 2251 raft_consensus.cc:2743] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:58.466742 1662 raft_consensus.cc:2388] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:58.456116 2339 raft_consensus.cc:3049] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:58.472282 2386 raft_consensus.cc:2798] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:58.490907 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:58.497530 1731 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.466617 1655 raft_consensus.cc:1235] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 3. Ops: []
I20250114 02:53:58.464686 2360 raft_consensus.cc:2798] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:58.469825 2350 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:58.500125 1553 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:58.503516 2350 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:58.503911 2350 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.504021 2099 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.505154 1729 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:58.506212 1729 raft_consensus.cc:3049] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:58.506516 1729 raft_consensus.cc:738] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:58.472437 2341 raft_consensus.cc:2743] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 02:53:58.509804 1731 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:58.477973 1702 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:58.469691 2412 raft_consensus.cc:2743] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:53:58.477429 2339 raft_consensus.cc:738] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:58.509101 1735 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:58.464198 1736 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
W20250114 02:53:58.532420 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 54510.3us
I20250114 02:53:58.470096 2303 raft_consensus.cc:2798] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:58.533174 1628 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.537714 1702 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:58.541069 1702 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.543526 1702 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.502350 2346 raft_consensus.cc:2798] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:58.546221 2346 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:58.546708 2346 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.530706 2385 raft_consensus.cc:2798] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:58.499652 2416 raft_consensus.cc:2743] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:58.507357 2386 raft_consensus.cc:695] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:58.555938 2412 raft_consensus.cc:2798] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:58.562588 2385 raft_consensus.cc:695] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:58.482429 2348 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.562817 1729 consensus_queue.cc:260] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.528908 2360 raft_consensus.cc:695] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:58.573388 1628 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.501703 1733 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:58.536958 2326 raft_consensus.cc:2743] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:53:58.507062 2251 consensus_queue.cc:1046] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20250114 02:53:58.537140 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:58.555433 2341 raft_consensus.cc:2798] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:58.593401 1628 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.499652 2246 consensus_queue.cc:1046] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.073s
I20250114 02:53:58.536470 2284 consensus_queue.cc:1046] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20250114 02:53:58.571173 2319 raft_consensus.cc:2798] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:58.554862 2303 raft_consensus.cc:695] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:58.572360 2412 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:58.575373 2307 consensus_queue.cc:1046] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20250114 02:53:58.620808 2341 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:53:58.601344 1728 raft_consensus.cc:2463] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
W20250114 02:53:58.628223 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 71407.5us
I20250114 02:53:58.662012 2368 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.663985 2368 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.664160 2380 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.660333 2385 consensus_queue.cc:237] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.665658 2248 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.667564 2248 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.660290 2412 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
W20250114 02:53:58.671032 2426 log.cc:926] Time spent T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.108s user 0.002s sys 0.000s
I20250114 02:53:58.674919 2246 raft_consensus.cc:3049] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:58.675022 2303 consensus_queue.cc:237] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.675632 2246 raft_consensus.cc:738] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:58.676739 2246 consensus_queue.cc:260] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.678401 2246 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.681807 2341 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.660029 2319 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:58.620388 2302 consensus_queue.cc:1046] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20250114 02:53:58.600247 1729 raft_consensus.cc:3054] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.688130 2433 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:58.688632 2433 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.688547 2360 consensus_queue.cc:237] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.599608 2339 consensus_queue.cc:260] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.689831 2363 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:58.690326 2363 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.686760 2319 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.689069 2248 raft_consensus.cc:3049] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:58.599673 2099 raft_consensus.cc:2798] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:58.621320 2379 raft_consensus.cc:3049] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:58.653198 2229 raft_consensus.cc:2798] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
W20250114 02:53:58.703580 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 139000us
W20250114 02:53:58.707893 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 46763.3us
I20250114 02:53:58.708560 1702 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.689854 2339 raft_consensus.cc:3054] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.693331 2248 raft_consensus.cc:738] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:58.681397 2412 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.725513 2440 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:58.693894 2433 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:58.725925 2440 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:58.726296 2440 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.726208 2433 consensus_queue.cc:1046] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:58.727249 2433 raft_consensus.cc:3049] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:58.727566 2433 raft_consensus.cc:738] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:58.728271 2433 consensus_queue.cc:260] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.729326 2433 raft_consensus.cc:3054] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.730998 2248 consensus_queue.cc:260] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.732913 2440 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.692668 2197 raft_consensus.cc:2798] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:58.716243 1580 raft_consensus.cc:2388] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:58.734097 2197 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:58.734274 2262 raft_consensus.cc:2743] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:53:58.734637 2197 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.717357 2380 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.719563 2242 raft_consensus.cc:3054] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.721395 2348 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.722921 2323 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.698951 2379 raft_consensus.cc:738] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:58.693260 2099 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:58.670337 2346 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.744386 2439 consensus_queue.cc:1046] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:53:58.715011 2229 raft_consensus.cc:695] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:58.591733 2416 consensus_queue.cc:1046] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.124s
I20250114 02:53:58.704114 2387 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.756174 2346 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.757622 2387 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.731606 2412 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:58.734849 2440 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:58.742640 2441 consensus_queue.cc:1046] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20250114 02:53:58.711694 2326 consensus_queue.cc:1046] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20250114 02:53:58.655915 2386 consensus_queue.cc:237] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.735266 2307 raft_consensus.cc:3049] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:58.748165 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.748778 2442 consensus_queue.cc:1046] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20250114 02:53:58.749106 2229 consensus_queue.cc:237] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.750773 2323 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.751546 2363 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:58.744956 2379 consensus_queue.cc:260] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:58.737080 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 143763us
I20250114 02:53:58.745458 2099 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.761961 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.788612 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.777473 1657 raft_consensus.cc:1270] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:57.303946 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:58.780911 2197 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.740191 2350 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.771992 2307 raft_consensus.cc:738] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:58.805708 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.815093 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.845490 1628 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.828914 1660 raft_consensus.cc:2363] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3.
W20250114 02:53:58.828092 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 89662.9us
I20250114 02:53:58.826320 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:58.829397 1661 raft_consensus.cc:2388] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:58.854830 1628 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.786235 2412 consensus_queue.cc:1046] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20250114 02:53:58.859169 1661 raft_consensus.cc:1235] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20250114 02:53:58.868414 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:53:58.867877 1661 raft_consensus.cc:1235] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:53:58.865599 1628 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.827203 2379 raft_consensus.cc:3054] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.849025 2448 raft_consensus.cc:3049] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:58.872428 1661 raft_consensus.cc:1235] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:53:58.869657 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 288)
W20250114 02:53:58.825726 1732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 347) took 2009 ms (client timeout 2417 ms). Trace:
I20250114 02:53:58.872802 1628 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.874615 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 288)
I20250114 02:53:58.863891 2412 raft_consensus.cc:3049] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:58.856243 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.875962 1628 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.869555 2436 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:58.884172 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.873658 2448 raft_consensus.cc:738] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:58.861111 2437 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:58.873960 2428 raft_consensus.cc:2798] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:58.886878 1628 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:58.884606 2445 consensus_queue.cc:1046] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:53:58.887674 2323 consensus_queue.cc:1035] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:58.888113 2392 consensus_queue.cc:1046] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:53:58.888504 2350 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.886121 1661 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.880260 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 288)
I20250114 02:53:58.873262 2385 consensus_queue.cc:1046] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20250114 02:53:58.852265 2346 consensus_queue.cc:1035] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20250114 02:53:58.954172 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:58.954838 2416 consensus_queue.cc:1046] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:53:58.955798 2433 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:58.870734 2439 consensus_queue.cc:1046] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20250114 02:53:58.873874 2360 consensus_queue.cc:1046] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20250114 02:53:58.872370 2303 consensus_queue.cc:1035] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20250114 02:53:58.881868 2387 raft_consensus.cc:2798] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:58.956454 2445 raft_consensus.cc:3049] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
W20250114 02:53:58.880587 1732 rpcz_store.cc:269] 0114 02:53:56.815791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.303693 (+487902us) service_pool.cc:225] Handling call
0114 02:53:58.825648 (+1521955us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 02:53:58.960866 1550 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items [suppressed 19 similar messages]
I20250114 02:53:58.968845 2352 raft_consensus.cc:2798] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:58.882306 2412 raft_consensus.cc:738] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:59.039191 2412 consensus_queue.cc:260] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.040781 2412 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.972715 1661 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:53:59.044903 1661 raft_consensus.cc:1235] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:53:58.973003 1653 raft_consensus.cc:2388] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:58.889115 2437 raft_consensus.cc:695] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:59.047411 2437 consensus_queue.cc:237] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.050046 1653 raft_consensus.cc:1235] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.891393 2427 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:58.953078 2436 raft_consensus.cc:695] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:59.051287 2436 consensus_queue.cc:237] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.052003 2414 consensus_queue.cc:1046] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:53:59.053083 2414 consensus_queue.cc:1035] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:53:58.960865 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 72360.3us
I20250114 02:53:59.057974 1659 raft_consensus.cc:1235] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:58.983098 2439 raft_consensus.cc:3049] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:59.079869 2439 raft_consensus.cc:738] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:59.080718 2439 consensus_queue.cc:260] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.081995 2439 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:58.982652 2416 raft_consensus.cc:3049] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:58.875119 2446 consensus_queue.cc:1046] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.106s
W20250114 02:53:58.985018 1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 339) took 2101 ms (client timeout 2229 ms). Trace:
I20250114 02:53:59.082815 2416 raft_consensus.cc:738] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
W20250114 02:53:59.083036 1586 rpcz_store.cc:269] 0114 02:53:56.884910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.155265 (+270355us) service_pool.cc:225] Handling call
0114 02:53:58.984863 (+1829598us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b5400175054. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:58.984964 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":693196672}
I20250114 02:53:59.083637 2416 consensus_queue.cc:260] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.084165 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.084921 1586 raft_consensus.cc:2388] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:53:59.084889 2416 raft_consensus.cc:3054] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:53:59.085458 1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250) took 1540 ms (client timeout 2021 ms). Trace:
W20250114 02:53:59.085790 1586 rpcz_store.cc:269] 0114 02:53:57.544609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.083924 (+1539315us) service_pool.cc:225] Handling call
0114 02:53:59.085407 (+ 1483us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:58.990067 1729 raft_consensus.cc:1270] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250114 02:53:59.086745 1729 pending_rounds.cc:77] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 0
I20250114 02:53:59.087020 1729 pending_rounds.cc:99] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250114 02:53:59.087273 1729 raft_consensus.cc:2883] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:53:59.087348 2268 consensus_queue.cc:1046] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:59.088543 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:59.089390 1729 raft_consensus.cc:2388] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:59.090698 2452 raft_consensus.cc:3049] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:59.091081 2452 raft_consensus.cc:738] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:59.091847 2452 consensus_queue.cc:260] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.093019 2452 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.093497 2427 raft_consensus.cc:695] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:58.990959 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:53:58.991629 1578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 323) took 2198 ms (client timeout 1645 ms). Trace:
I20250114 02:53:58.979787 2392 raft_consensus.cc:3049] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:58.964753 2099 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:58.979741 2445 raft_consensus.cc:738] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:58.986753 2197 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:58.884917 2447 consensus_queue.cc:1046] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.095s
I20250114 02:53:58.986776 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:59.168648 2447 raft_consensus.cc:3049] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:59.169045 2447 raft_consensus.cc:738] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:58.982612 2360 raft_consensus.cc:3049] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:59.171633 2360 raft_consensus.cc:738] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:59.172474 2360 consensus_queue.cc:260] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.173717 2360 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.985615 2385 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:58.977516 2430 raft_consensus.cc:3049] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:58.974129 2433 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.176352 2430 raft_consensus.cc:738] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:59.176406 2385 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.177280 2430 consensus_queue.cc:260] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.004696 1483 catalog_manager.cc:5526] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 2, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:59.178383 2430 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.179314 1483 catalog_manager.cc:5526] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 2, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:59.180967 1483 catalog_manager.cc:5526] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:59.016652 2248 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.012821 2442 raft_consensus.cc:3049] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:59.182804 2442 raft_consensus.cc:738] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:59.183646 2442 consensus_queue.cc:260] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.184834 2442 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.017966 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 338) took 2137 ms. Trace:
I20250114 02:53:59.186686 1584 rpcz_store.cc:276] 0114 02:53:56.880548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.191687 (+311139us) service_pool.cc:225] Handling call
0114 02:53:57.192008 (+ 321us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:53:58.221193 (+1029185us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:53:58.221225 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:53:58.221637 (+ 412us) log.cc:844] Serialized 26 byte log entry
0114 02:53:58.299533 (+ 77896us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:53:58.429762 (+130229us) log.cc:844] Serialized 14 byte log entry
0114 02:53:58.439284 (+ 9522us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:59.017481 (+578197us) spinlock_profiling.cc:243] Waited 68.8 ms on lock 0x7b5400291b54. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:53:59.017537 (+ 56us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:53:59.017594 (+ 57us) raft_consensus.cc:1680] finished
0114 02:53:59.017706 (+ 112us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:53:59.017898 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":195538,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":21609,"raft.queue_time_us":1154,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":152,"spinlock_wait_cycles":158156160,"thread_start_us":184965,"threads_started":2,"wal-append.queue_time_us":133385}
I20250114 02:53:59.189201 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
W20250114 02:53:59.189966 1584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362) took 1613 ms (client timeout 2073 ms). Trace:
I20250114 02:53:58.979995 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 288)
I20250114 02:53:59.063583 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:59.195044 1732 raft_consensus.cc:2388] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:59.195895 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.196503 1732 raft_consensus.cc:3054] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.063868 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.198176 1728 raft_consensus.cc:2371] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1. Re-sending same reply.
I20250114 02:53:59.209923 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.210621 1728 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:58.880470 2307 consensus_queue.cc:260] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.212239 2307 raft_consensus.cc:3054] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.066855 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:59.215484 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.216084 1736 raft_consensus.cc:3054] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.068255 1733 raft_consensus.cc:1235] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20250114 02:53:59.219264 1587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 368) took 1324 ms (client timeout 1538 ms). Trace:
I20250114 02:53:59.219439 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
W20250114 02:53:59.219662 1587 rpcz_store.cc:269] 0114 02:53:57.895171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:58.230421 (+335250us) service_pool.cc:225] Handling call
0114 02:53:59.219105 (+988684us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b540016c954. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:59.219208 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9407232}
I20250114 02:53:59.069137 1735 raft_consensus.cc:1235] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:53:59.220690 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.221300 1733 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.221237 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.222384 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.222944 1735 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
W20250114 02:53:59.224038 1580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 325) took 2430 ms (client timeout 2777 ms). Trace:
W20250114 02:53:59.224470 1580 rpcz_store.cc:269] 0114 02:53:56.793869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:56.797059 (+ 3190us) service_pool.cc:225] Handling call
0114 02:53:58.391668 (+1594609us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:53:59.223827 (+832159us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b5400174b54. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:59.223967 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":709431936}
I20250114 02:53:59.070780 1731 raft_consensus.cc:1235] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250114 02:53:59.077531 2414 raft_consensus.cc:3049] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:59.226786 2414 raft_consensus.cc:738] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:59.227327 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.228091 1731 raft_consensus.cc:2388] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:59.227577 2414 consensus_queue.cc:260] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.228619 2414 raft_consensus.cc:3054] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.228838 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.229593 1731 raft_consensus.cc:2388] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:59.079283 1654 raft_consensus.cc:1235] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 3. Ops: []
I20250114 02:53:59.083356 2446 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:58.992755 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 242) took 2631 ms. Trace:
I20250114 02:53:59.094957 1662 raft_consensus.cc:2463] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:53:59.230926 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 271) took 2232 ms. Trace:
I20250114 02:53:59.230768 2446 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.231537 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.231987 1662 raft_consensus.cc:1270] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:59.178483 2385 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:59.086598 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:59.232800 1731 raft_consensus.cc:1235] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:53:59.232944 1586 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.233084 2446 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:59.098353 1702 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
W20250114 02:53:59.233880 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 135587us
I20250114 02:53:59.209964 2363 raft_consensus.cc:3049] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:59.234704 2363 raft_consensus.cc:738] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:59.235483 2363 consensus_queue.cc:260] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.236524 2363 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.090173 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.237978 2442 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:59.239408 1729 raft_consensus.cc:1235] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20250114 02:53:59.096155 1578 rpcz_store.cc:269] 0114 02:53:56.793188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:56.797007 (+ 3819us) service_pool.cc:225] Handling call
0114 02:53:58.991424 (+2194417us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b540033bdd4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:58.991550 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3755396992}
I20250114 02:53:59.096424 2392 raft_consensus.cc:738] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:59.094343 2427 consensus_queue.cc:237] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.093042 2387 raft_consensus.cc:695] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:59.075116 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:59.105849 2352 raft_consensus.cc:695] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:59.109872 2440 raft_consensus.cc:3049] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:59.110988 2319 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.115753 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.121660 1628 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.130746 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.136880 1660 raft_consensus.cc:1235] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:53:59.138427 1553 leader_election.cc:400] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3.
I20250114 02:53:59.140254 2445 consensus_queue.cc:260] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.247520 1553 leader_election.cc:403] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250114 02:53:59.248000 2352 consensus_queue.cc:237] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.143638 2099 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:59.149639 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.231379 1654 rpcz_store.cc:276] 0114 02:53:56.998354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.312083 (+313729us) service_pool.cc:225] Handling call
0114 02:53:59.079113 (+1767030us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:59.230627 (+151514us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:59.230869 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.248862 2387 consensus_queue.cc:237] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.249580 1579 raft_consensus.cc:2388] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
W20250114 02:53:59.250025 1579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 266) took 1679 ms (client timeout 1656 ms). Trace:
I20250114 02:53:59.154871 2441 raft_consensus.cc:3049] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:59.250958 2441 raft_consensus.cc:738] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:59.240768 2392 consensus_queue.cc:260] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.251502 1581 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
W20250114 02:53:59.252025 1581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 283) took 1649 ms (client timeout 1900 ms). Trace:
I20250114 02:53:59.252226 2392 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.251917 2441 consensus_queue.cc:260] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:59.252465 1581 rpcz_store.cc:269] 0114 02:53:57.602987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.130325 (+1527338us) service_pool.cc:225] Handling call
0114 02:53:59.251968 (+121643us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.253062 2441 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.253505 2414 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.254130 2445 raft_consensus.cc:3054] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.156040 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:59.254724 1734 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.156667 1659 raft_consensus.cc:1235] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
W20250114 02:53:59.255725 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 134172us
I20250114 02:53:59.256013 2440 raft_consensus.cc:738] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:59.256692 1628 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:59.256865 2416 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:59.257511 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:59.257380 2416 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.257694 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:59.257969 1628 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.258471 1658 raft_consensus.cc:2388] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:59.258693 2357 raft_consensus.cc:2743] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:59.258726 2099 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:59.259315 2357 raft_consensus.cc:3054] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.259424 2416 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:58.979468 2348 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:59.230692 1656 rpcz_store.cc:276] 0114 02:53:56.361181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:56.976572 (+615391us) service_pool.cc:225] Handling call
0114 02:53:56.980755 (+ 4183us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:53:57.240808 (+260053us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:53:58.992520 (+1751712us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54001d6ad4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:53:58.992694 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":430683392}
W20250114 02:53:59.250865 1579 rpcz_store.cc:269] 0114 02:53:57.570823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.149355 (+1578532us) service_pool.cc:225] Handling call
0114 02:53:59.249959 (+100604us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.240267 2412 consensus_queue.cc:1046] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20250114 02:53:58.998919 2448 consensus_queue.cc:260] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.261336 1660 raft_consensus.cc:1235] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:53:59.261682 2448 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.177803 2433 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:58.971767 2428 raft_consensus.cc:695] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:57.361500 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.263290 1730 raft_consensus.cc:3054] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.208508 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.240347 2442 raft_consensus.cc:695] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:53:59.260514 1702 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:59.192118 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 293)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 294)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 295)
I20250114 02:53:59.265293 1628 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:59.266315 1628 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.259428 2099 raft_consensus.cc:695] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:59.260480 2348 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.169991 2447 consensus_queue.cc:260] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.165294 2251 raft_consensus.cc:3049] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:59.250186 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:59.264657 1585 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:53:59.267230 1735 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
W20250114 02:53:59.248070 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 109859us
W20250114 02:53:59.269271 1585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 273) took 1692 ms (client timeout 1959 ms). Trace:
W20250114 02:53:59.269600 1585 rpcz_store.cc:269] 0114 02:53:57.581361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.208221 (+1626860us) service_pool.cc:225] Handling call
0114 02:53:59.269216 (+ 60995us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.266180 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 293)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 294)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 295)
I20250114 02:53:59.272142 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 293)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 294)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 298)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 299)
I20250114 02:53:59.182408 1483 catalog_manager.cc:5526] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 2, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:59.275496 1483 catalog_manager.cc:5526] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 2, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:59.274204 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 293)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 294)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 298)
I20250114 02:53:59.278946 1662 raft_consensus.cc:1235] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:59.277494 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 293)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 294)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 295)
I20250114 02:53:59.282276 1553 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.280879 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 293)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 294)
I20250114 02:53:59.283712 1483 catalog_manager.cc:5526] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 2, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:53:59.285528 1586 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
W20250114 02:53:59.286185 1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 358) took 1710 ms (client timeout 2051 ms). Trace:
I20250114 02:53:59.254935 1583 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.263615 2428 consensus_queue.cc:237] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.143605 1653 raft_consensus.cc:1235] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250114 02:53:59.284600 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 293)
W20250114 02:53:59.286654 1586 rpcz_store.cc:269] 0114 02:53:57.575415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.086399 (+1510984us) service_pool.cc:225] Handling call
0114 02:53:59.286116 (+199717us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.291349 2348 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:59.248725 2319 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:59.297281 1628 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:59.299825 1553 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.300650 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.256870 2440 consensus_queue.cc:260] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.301749 2440 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.301832 2416 raft_consensus.cc:2798] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250114 02:53:59.190356 1584 rpcz_store.cc:269] 0114 02:53:57.576543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.188895 (+1612352us) service_pool.cc:225] Handling call
0114 02:53:59.189907 (+ 1012us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.268858 2197 raft_consensus.cc:2798] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:53:59.306654 2099 consensus_queue.cc:237] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.304420 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:59.312378 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:59.312810 1660 raft_consensus.cc:2388] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:59.278512 2251 raft_consensus.cc:738] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:59.288482 2229 raft_consensus.cc:2743] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:59.310906 2348 consensus_queue.cc:1046] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:53:59.308092 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 288)
I20250114 02:53:59.313974 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:59.288486 2430 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:59.295034 2303 raft_consensus.cc:2743] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:53:59.320535 1585 raft_consensus.cc:3054] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.321278 1734 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:59.322371 1628 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:59.322444 1553 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.323120 2365 raft_consensus.cc:2798] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:59.323307 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.323588 2365 raft_consensus.cc:695] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:59.324676 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:53:59.324338 2365 consensus_queue.cc:237] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.325556 1628 leader_election.cc:304] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.327646 1585 raft_consensus.cc:2463] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:59.302273 2416 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:59.292654 2436 consensus_queue.cc:1035] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20250114 02:53:59.307469 1583 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:59.366329 1736 raft_consensus.cc:2463] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:59.363658 2303 raft_consensus.cc:2743] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:53:59.352110 2106 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:59.371623 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.388353 1587 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
W20250114 02:53:59.388948 1587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 279) took 1787 ms (client timeout 2114 ms). Trace:
W20250114 02:53:59.389364 1587 rpcz_store.cc:269] 0114 02:53:57.601843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.371375 (+1769532us) service_pool.cc:225] Handling call
0114 02:53:59.388885 (+ 17510us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 02:53:59.370332 1585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 371) took 1396 ms (client timeout 1749 ms). Trace:
I20250114 02:53:59.312646 2339 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:59.364195 1553 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.329496 2416 raft_consensus.cc:3054] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.357139 2302 raft_consensus.cc:3049] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:53:59.369697 1734 raft_consensus.cc:2388] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:59.363730 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.329466 2430 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:59.379362 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 89903.6us
W20250114 02:53:59.319481 2341 consensus_meta.cc:287] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.635s user 0.003s sys 0.010s
I20250114 02:53:59.312274 2447 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.361883 2242 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:59.383606 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 288)
I20250114 02:53:59.391886 1553 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.400267 1732 raft_consensus.cc:2463] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:59.404994 1735 raft_consensus.cc:1235] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:53:59.394534 2323 raft_consensus.cc:2743] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:53:59.340646 2229 raft_consensus.cc:2743] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:59.329006 2385 raft_consensus.cc:2743] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:59.320855 2429 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:59.320715 2197 raft_consensus.cc:695] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
W20250114 02:53:59.394548 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 69086.4us
I20250114 02:53:59.366677 2251 consensus_queue.cc:260] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.369695 2442 consensus_queue.cc:237] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.399847 2341 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.403970 2242 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.407181 1702 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:53:59.394109 2106 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.410033 2429 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.415549 2251 raft_consensus.cc:3054] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.416409 2106 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:59.395423 2302 raft_consensus.cc:738] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
W20250114 02:53:59.392436 1585 rpcz_store.cc:269] 0114 02:53:57.974296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.288367 (+1314071us) service_pool.cc:225] Handling call
0114 02:53:59.370269 (+ 81902us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 02:53:59.394140 1583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 357) took 1894 ms (client timeout 2023 ms). Trace:
I20250114 02:53:59.403049 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262)
I20250114 02:53:59.413321 1628 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:59.417397 2429 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:59.401573 2339 raft_consensus.cc:695] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:59.418143 1702 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:59.418438 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:59.404110 2248 raft_consensus.cc:2743] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:53:59.421260 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:59.403543 1584 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
W20250114 02:53:59.424659 1584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 379) took 1229 ms (client timeout 1622 ms). Trace:
W20250114 02:53:59.425000 1702 consensus_peers.cc:487] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 -> Peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Couldn't send request to peer 62daa58e8ff64698bb46ea3fd5290cdd. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
W20250114 02:53:59.425057 1584 rpcz_store.cc:269] 0114 02:53:58.194793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.328777 (+1133984us) service_pool.cc:225] Handling call
0114 02:53:59.424602 (+ 95825us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.425638 1702 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:53:59.423939 1550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 262) took 1855 ms. Trace:
I20250114 02:53:59.426152 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:59.426203 1550 rpcz_store.cc:276] 0114 02:53:57.568908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.423893 (+1854985us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250114 02:53:59.426445 2441 raft_consensus.cc:2758] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250114 02:53:59.426875 1584 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
W20250114 02:53:59.427433 1702 consensus_peers.cc:487] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 -> Peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Couldn't send request to peer 62daa58e8ff64698bb46ea3fd5290cdd. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:59.422628 1628 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:59.431357 2445 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:59.432726 2229 raft_consensus.cc:2798] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:59.433266 2229 raft_consensus.cc:695] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:59.434145 2229 consensus_queue.cc:237] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.418916 2440 raft_consensus.cc:2743] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:53:59.428274 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
I20250114 02:53:59.435879 2197 consensus_queue.cc:237] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.439158 2430 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:59.443423 1553 leader_election.cc:304] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.440430 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
I20250114 02:53:59.413913 2379 consensus_queue.cc:1035] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20250114 02:53:59.417413 2302 consensus_queue.cc:260] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.453172 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
I20250114 02:53:59.437301 2428 raft_consensus.cc:2798] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:59.444514 2248 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:59.460578 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259)
I20250114 02:53:59.465966 1550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 259) took 1912 ms. Trace:
I20250114 02:53:59.466006 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:59.466508 1550 rpcz_store.cc:276] 0114 02:53:57.553391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.465924 (+1912533us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250114 02:53:59.467502 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 402)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
I20250114 02:53:59.458020 2385 raft_consensus.cc:2743] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:59.470656 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:59.471494 1656 raft_consensus.cc:2363] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3.
I20250114 02:53:59.471634 2414 raft_consensus.cc:2743] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3.
I20250114 02:53:59.472864 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.473728 1578 raft_consensus.cc:2388] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
W20250114 02:53:59.474185 1578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 260) took 1917 ms (client timeout 2198 ms). Trace:
I20250114 02:53:59.471246 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 402)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
I20250114 02:53:59.475275 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:59.476101 1581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 255) took 1928 ms (client timeout 2221 ms). Trace:
W20250114 02:53:59.476444 1581 rpcz_store.cc:269] 0114 02:53:57.547895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.475011 (+1927116us) service_pool.cc:225] Handling call
0114 02:53:59.476047 (+ 1036us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.477135 1730 raft_consensus.cc:2463] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
W20250114 02:53:59.477672 1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 337) took 2686 ms (client timeout 1529 ms). Trace:
W20250114 02:53:59.478041 1730 rpcz_store.cc:269] 0114 02:53:56.790750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:57.361119 (+570369us) service_pool.cc:225] Handling call
0114 02:53:59.477580 (+2116461us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.475183 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 402)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
I20250114 02:53:59.479629 1586 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:59.469720 1728 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:59.479897 1553 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.483811 1734 raft_consensus.cc:1235] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:53:59.484709 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:59.480983 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 402)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253)
I20250114 02:53:59.485319 1734 raft_consensus.cc:2363] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3.
I20250114 02:53:59.485986 1550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 253) took 1939 ms. Trace:
I20250114 02:53:59.486322 1550 rpcz_store.cc:276] 0114 02:53:57.546635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.485941 (+1939306us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250114 02:53:59.487103 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:59.488766 2386 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.006s sys 0.009s
W20250114 02:53:59.489117 2386 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s user 0.007s sys 0.010s
I20250114 02:53:59.475107 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:59.490322 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:59.491027 1658 raft_consensus.cc:2463] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
W20250114 02:53:59.491576 1582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 326) took 2697 ms (client timeout 1687 ms). Trace:
I20250114 02:53:59.467161 2440 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:59.492317 2242 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:59.486311 1734 raft_consensus.cc:1270] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250114 02:53:59.492657 2440 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.491086 1654 raft_consensus.cc:1270] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250114 02:53:59.420742 1583 rpcz_store.cc:269] 0114 02:53:57.499722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.155742 (+1656020us) service_pool.cc:225] Handling call
0114 02:53:59.393459 (+237717us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.463708 1727 raft_consensus.cc:2388] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:59.497335 1731 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:59.478667 2248 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.487015 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 402)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252)
W20250114 02:53:59.553714 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 70224.6us
W20250114 02:53:59.554020 1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 372) took 1577 ms (client timeout 1803 ms). Trace:
I20250114 02:53:59.554307 1550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 252) took 2008 ms. Trace:
I20250114 02:53:59.554450 1553 leader_election.cc:400] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3.
W20250114 02:53:59.554283 1586 rpcz_store.cc:269] 0114 02:53:57.976992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.300431 (+1323439us) service_pool.cc:225] Handling call
0114 02:53:59.553902 (+253471us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b540012e154. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:59.553979 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":410085248}
I20250114 02:53:59.554937 1553 leader_election.cc:403] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250114 02:53:59.555279 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.555927 1553 leader_election.cc:304] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:59.555958 1586 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
W20250114 02:53:59.492033 1582 rpcz_store.cc:269] 0114 02:53:56.794190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:56.970997 (+176807us) service_pool.cc:225] Handling call
0114 02:53:59.491383 (+2520386us) spinlock_profiling.cc:243] Waited 2.46 s on lock 0x7b540022f454. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:59.491513 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5662216960}
I20250114 02:53:59.496729 1732 raft_consensus.cc:1270] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:59.506636 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.455315 2268 consensus_queue.cc:1046] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20250114 02:53:59.511508 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:59.496088 1733 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
W20250114 02:53:59.549938 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 51420.3us
I20250114 02:53:59.562708 1553 leader_election.cc:400] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3.
I20250114 02:53:59.431247 2339 consensus_queue.cc:237] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.554745 1550 rpcz_store.cc:276] 0114 02:53:57.545693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.554261 (+2008568us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250114 02:53:59.474521 1578 rpcz_store.cc:269] 0114 02:53:57.557207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.472516 (+1915309us) service_pool.cc:225] Handling call
0114 02:53:59.474130 (+ 1614us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.522751 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:59.567201 1587 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.561098 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.568223 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 386) took 1247 ms. Trace:
I20250114 02:53:59.568565 1583 rpcz_store.cc:276] 0114 02:53:58.321260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.506482 (+1185222us) service_pool.cc:225] Handling call
0114 02:53:59.568164 (+ 61682us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.566903 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34938 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 390)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 402)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 394)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 409)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
I20250114 02:53:59.491611 2302 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.556895 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 382) took 1321 ms. Trace:
I20250114 02:53:59.499830 2385 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.553267 2319 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:59.559051 2242 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.533517 1584 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:59.521934 2433 raft_consensus.cc:3049] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:59.550844 2441 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:53:59.577105 1586 rpcz_store.cc:276] 0114 02:53:58.235309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.555066 (+1319757us) service_pool.cc:225] Handling call
0114 02:53:59.556856 (+ 1790us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.501343 2428 raft_consensus.cc:695] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:59.556866 2440 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:59.562760 1736 raft_consensus.cc:2463] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:59.561268 2106 raft_consensus.cc:2798] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:59.559613 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:53:59.646911 1584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 365) took 2043 ms (client timeout 2246 ms). Trace:
W20250114 02:53:59.647318 1584 rpcz_store.cc:269] 0114 02:53:57.603929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.425876 (+1821947us) service_pool.cc:225] Handling call
0114 02:53:59.646843 (+220967us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.648236 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:53:59.635046 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 63661.9us
I20250114 02:53:59.641240 1587 raft_consensus.cc:2463] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:59.600879 1553 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
W20250114 02:53:59.650779 1587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 366) took 2046 ms (client timeout 2213 ms). Trace:
W20250114 02:53:59.651119 1587 rpcz_store.cc:269] 0114 02:53:57.604162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.509064 (+1904902us) service_pool.cc:225] Handling call
0114 02:53:59.650722 (+141658us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.565457 2246 consensus_queue.cc:1035] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20250114 02:53:59.651988 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.652530 1587 raft_consensus.cc:2463] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:53:59.652627 2246 raft_consensus.cc:2798] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election won for term 2
W20250114 02:53:59.652953 1587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 374) took 1631 ms (client timeout 2169 ms). Trace:
I20250114 02:53:59.653177 2246 raft_consensus.cc:695] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
W20250114 02:53:59.653291 1587 rpcz_store.cc:269] 0114 02:53:58.022098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.651767 (+1629669us) service_pool.cc:225] Handling call
0114 02:53:59.652905 (+ 1138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.654122 2246 consensus_queue.cc:237] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.639272 2319 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:59.657001 2467 log.cc:926] Time spent T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
W20250114 02:53:59.651365 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 50582.6us
I20250114 02:53:59.658457 2319 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:59.654220 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.558519 2357 raft_consensus.cc:2743] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:53:59.651365 1628 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:59.618124 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:59.643219 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.645391 2106 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:59.644259 2433 raft_consensus.cc:738] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
W20250114 02:53:59.646399 1581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 384) took 1404 ms (client timeout 1645 ms). Trace:
I20250114 02:53:59.577118 2429 consensus_queue.cc:1035] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.084s
I20250114 02:53:59.660709 2341 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:59.662223 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:53:59.660595 1730 raft_consensus.cc:2463] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
W20250114 02:53:59.711169 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 59934.2us
I20250114 02:53:59.646023 2441 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:59.713444 2466 log.cc:926] Time spent T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20250114 02:53:59.714879 2469 log.cc:926] Time spent T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
I20250114 02:53:59.700681 1582 raft_consensus.cc:2463] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:53:59.716732 2445 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
W20250114 02:53:59.717290 1582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 254) took 2170 ms (client timeout 2499 ms). Trace:
I20250114 02:53:59.718890 1733 raft_consensus.cc:2388] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:53:59.719683 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:59.720280 1733 raft_consensus.cc:1270] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:53:59.720852 1583 raft_consensus.cc:2405] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250114 02:53:59.721381 1583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 381) took 1486 ms (client timeout 1862 ms). Trace:
I20250114 02:53:59.722992 2197 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:59.723297 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 406) took 1414 ms. Trace:
I20250114 02:53:59.723516 2197 raft_consensus.cc:695] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:59.723695 1727 rpcz_store.cc:276] 0114 02:53:58.309045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:58.468276 (+159231us) service_pool.cc:225] Handling call
0114 02:53:59.723116 (+1254840us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b54002cbfd4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:59.723243 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1939191552}
I20250114 02:53:59.724481 2197 consensus_queue.cc:237] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.726994 1735 raft_consensus.cc:1235] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:53:59.727300 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
W20250114 02:53:59.728199 1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 377) took 1567 ms (client timeout 1959 ms). Trace:
I20250114 02:53:59.697703 2106 raft_consensus.cc:3054] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
W20250114 02:53:59.728631 1586 rpcz_store.cc:269] 0114 02:53:58.161251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.639857 (+1478606us) service_pool.cc:225] Handling call
0114 02:53:59.727139 (+ 87282us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:53:59.728137 (+ 998us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8071040}
I20250114 02:53:59.690785 2379 consensus_queue.cc:1035] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20250114 02:53:59.703878 1732 raft_consensus.cc:1235] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
W20250114 02:53:59.721848 1583 rpcz_store.cc:269] 0114 02:53:58.235028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.578612 (+1343584us) service_pool.cc:225] Handling call
0114 02:53:59.719498 (+140886us) spinlock_profiling.cc:243] Waited 3.97 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:53:59.721314 (+ 1816us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9134848}
I20250114 02:53:59.735518 1727 raft_consensus.cc:1270] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:53:59.737198 1657 raft_consensus.cc:1235] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:53:59.739660 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.727972 1658 raft_consensus.cc:2388] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:59.752257 1661 raft_consensus.cc:1235] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20250114 02:53:59.717741 1582 rpcz_store.cc:269] 0114 02:53:57.547196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.580560 (+2033364us) service_pool.cc:225] Handling call
0114 02:53:59.717229 (+136669us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.717314 2445 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.682024 2428 consensus_queue.cc:237] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:59.767786 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 56024.4us
W20250114 02:53:59.701781 1581 rpcz_store.cc:269] 0114 02:53:58.238829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.479120 (+1240291us) service_pool.cc:225] Handling call
0114 02:53:59.643038 (+163918us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.763353 2248 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:59.771193 2414 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:59.775502 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
W20250114 02:53:59.787194 2360 consensus_meta.cc:287] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.613s user 0.005s sys 0.009s
W20250114 02:53:59.787042 2363 consensus_meta.cc:287] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.550s user 0.011s sys 0.002s
I20250114 02:53:59.781440 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:59.830060 1628 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
W20250114 02:53:59.835276 1628 consensus_peers.cc:487] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 -> Peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Couldn't send request to peer 62daa58e8ff64698bb46ea3fd5290cdd. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
W20250114 02:53:59.835816 1628 consensus_peers.cc:487] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 -> Peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Couldn't send request to peer 62daa58e8ff64698bb46ea3fd5290cdd. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
W20250114 02:53:59.839372 1628 consensus_peers.cc:487] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 -> Peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Couldn't send request to peer 62daa58e8ff64698bb46ea3fd5290cdd. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:59.771179 2433 consensus_queue.cc:260] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:59.840116 1628 consensus_peers.cc:487] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 -> Peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Couldn't send request to peer 62daa58e8ff64698bb46ea3fd5290cdd. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:59.848047 1736 raft_consensus.cc:1270] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:53:59.837249 2360 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:59.831611 1581 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.833853 2242 raft_consensus.cc:2743] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3.
W20250114 02:53:59.850754 1628 consensus_peers.cc:487] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 -> Peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): Couldn't send request to peer 62daa58e8ff64698bb46ea3fd5290cdd. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:53:59.851509 1736 raft_consensus.cc:1235] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: []
I20250114 02:53:59.852115 2303 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:59.854699 2303 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.855044 1587 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:53:59.856663 2303 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:59.857136 2476 log.cc:926] Time spent T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20250114 02:53:59.837746 2427 raft_consensus.cc:2798] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250114 02:53:59.860929 1579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 361) took 2283 ms (client timeout 2094 ms). Trace:
I20250114 02:53:59.850512 1580 raft_consensus.cc:2363] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3.
I20250114 02:53:59.827656 2414 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.842295 2251 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:59.868604 2251 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.870236 2448 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:59.870453 1736 raft_consensus.cc:1270] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:53:59.870609 2251 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:59.871608 2448 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.873546 2448 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:59.873728 2414 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:59.860906 1583 raft_consensus.cc:2388] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
W20250114 02:53:59.874883 1583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 359) took 2299 ms (client timeout 2660 ms). Trace:
W20250114 02:53:59.875301 1583 rpcz_store.cc:269] 0114 02:53:57.575658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.765934 (+2190276us) service_pool.cc:225] Handling call
0114 02:53:59.874828 (+108894us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.876250 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:59.877138 1583 raft_consensus.cc:2405] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:53:59.876464 2106 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:59.877560 1583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 375) took 1755 ms (client timeout 2215 ms). Trace:
W20250114 02:53:59.877835 1583 rpcz_store.cc:269] 0114 02:53:58.122876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.876000 (+1753124us) service_pool.cc:225] Handling call
0114 02:53:59.877514 (+ 1514us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.878118 1586 raft_consensus.cc:2363] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3.
I20250114 02:53:59.862109 1628 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:59.879794 2229 raft_consensus.cc:2798] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:59.880524 2229 raft_consensus.cc:695] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:59.880333 1628 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:59.881994 1628 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:53:59.883049 1628 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.885623 1628 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:53:59.886277 2429 raft_consensus.cc:2798] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:59.852785 2360 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.886687 2429 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:53:59.883316 2483 raft_consensus.cc:2798] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:53:59.887094 2429 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.887275 2483 raft_consensus.cc:695] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:53:59.888000 2360 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:59.863345 2441 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:59.888104 2483 consensus_queue.cc:237] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.889755 2464 consensus_queue.cc:1035] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250114 02:53:59.891527 2478 log.cc:926] Time spent T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250114 02:53:59.892369 1585 raft_consensus.cc:2405] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:53:59.884301 2352 raft_consensus.cc:2743] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:59.895376 2433 raft_consensus.cc:3054] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.895968 2416 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.861378 2427 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:59.898806 2427 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.898687 2429 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:59.866143 1579 rpcz_store.cc:269] 0114 02:53:57.576283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.849446 (+2273163us) service_pool.cc:212] Skipping call since client already timed out
0114 02:53:59.852737 (+ 3291us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250114 02:53:59.901461 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:59.886389 2365 raft_consensus.cc:2798] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:53:59.902328 1579 raft_consensus.cc:2405] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20250114 02:53:59.910831 1579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 360) took 2334 ms (client timeout 2949 ms). Trace:
I20250114 02:53:59.910535 2229 consensus_queue.cc:237] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:59.911240 1579 rpcz_store.cc:269] 0114 02:53:57.575977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.901183 (+2325206us) service_pool.cc:225] Handling call
0114 02:53:59.910772 (+ 9589us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.912158 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.913374 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:59.886417 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:53:59.918778 1628 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.900628 2429 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:59.902830 2385 raft_consensus.cc:2743] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:53:59.906849 2365 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:53:59.922127 2464 raft_consensus.cc:2743] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:53:59.923782 2365 raft_consensus.cc:3054] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:53:59.923365 1628 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:53:59.927250 2380 raft_consensus.cc:3049] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:53:59.927738 2380 raft_consensus.cc:738] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:53:59.928259 2464 raft_consensus.cc:2743] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:53:59.928633 2380 consensus_queue.cc:260] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.933205 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:59.933554 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:59.934621 1659 raft_consensus.cc:2388] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:53:59.935869 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:59.936619 1662 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:59.938918 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:53:59.939625 1655 raft_consensus.cc:3054] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:53:59.940183 1584 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:53:59.943239 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:53:59.943977 1658 raft_consensus.cc:2463] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:53:59.952865 1553 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:59.969529 1553 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:53:59.918996 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:53:59.923085 2106 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:53:59.962873 2302 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:53:59.954054 1583 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:53:59.966288 1581 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:53:59.974011 2251 raft_consensus.cc:2798] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250114 02:53:59.981603 2474 log.cc:926] Time spent T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.152s user 0.001s sys 0.001s
W20250114 02:53:59.972355 1584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 267) took 2401 ms (client timeout 2619 ms). Trace:
I20250114 02:53:59.976286 2414 raft_consensus.cc:2798] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:53:59.985811 2445 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:53:59.987531 2302 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:53:59.995112 1757 debug-util.cc:398] Leaking SignalData structure 0x7b080014a220 after lost signal to thread 428
I20250114 02:53:59.995404 2445 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:53:59.995517 2302 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:53:59.996052 1757 debug-util.cc:398] Leaking SignalData structure 0x7b08001f1480 after lost signal to thread 1519
W20250114 02:53:59.997259 1757 debug-util.cc:398] Leaking SignalData structure 0x7b08002ae0a0 after lost signal to thread 1611
W20250114 02:53:59.998248 1757 debug-util.cc:398] Leaking SignalData structure 0x7b08002ead00 after lost signal to thread 1686
I20250114 02:53:59.999034 1732 raft_consensus.cc:1270] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:53:59.999891 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:00.000574 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.001468 1578 raft_consensus.cc:2405] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250114 02:54:00.001895 1578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 380) took 1775 ms (client timeout 1884 ms). Trace:
W20250114 02:53:59.992218 1584 rpcz_store.cc:269] 0114 02:53:57.571069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.647989 (+2076920us) service_pool.cc:225] Handling call
0114 02:53:59.963687 (+315698us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b54000ca3d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:53:59.968270 (+ 4583us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":669887872}
I20250114 02:53:59.982578 1579 raft_consensus.cc:2371] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3. Re-sending same reply.
I20250114 02:54:00.009182 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.010342 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.011080 1584 raft_consensus.cc:2388] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:00.011833 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.012218 2380 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:00.012377 1584 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:00.012924 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 285) took 1666 ms. Trace:
I20250114 02:54:00.013504 2427 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:00.015059 1580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 265) took 2444 ms (client timeout 2209 ms). Trace:
W20250114 02:54:00.015610 1580 rpcz_store.cc:269] 0114 02:53:57.570525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.486865 (+1916340us) service_pool.cc:225] Handling call
0114 02:54:00.014866 (+528001us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b5400291654. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:00.014979 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":796530944}
I20250114 02:53:59.991539 2251 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 02:53:59.998509 1757 debug-util.cc:398] Leaking SignalData structure 0x7b08002eada0 after lost signal to thread 1687
W20250114 02:53:59.988077 1583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 258) took 2436 ms (client timeout 2940 ms). Trace:
W20250114 02:54:00.002249 1578 rpcz_store.cc:269] 0114 02:53:58.234725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.000297 (+1765572us) service_pool.cc:225] Handling call
0114 02:54:00.001841 (+ 1544us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 02:53:59.992422 1581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 385) took 1731 ms (client timeout 2051 ms). Trace:
I20250114 02:53:59.993377 2414 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:00.013228 1584 rpcz_store.cc:276] 0114 02:53:58.346675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.011587 (+1664912us) service_pool.cc:225] Handling call
0114 02:54:00.012877 (+ 1290us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:53:59.993566 1655 raft_consensus.cc:2463] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:00.016942 2416 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:00.016602 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.034260 2251 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:00.039336 2414 raft_consensus.cc:3054] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:00.054625 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:00.040871 2379 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.045153 2433 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:00.043327 1628 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:00.048086 2360 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.031574 2365 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.064239 1553 leader_election.cc:304] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:00.054922 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.067019 2350 raft_consensus.cc:2798] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:00.067598 1579 raft_consensus.cc:2405] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:00.067687 2302 raft_consensus.cc:2798] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election won for term 3
W20250114 02:54:00.037819 1583 rpcz_store.cc:269] 0114 02:53:57.551298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.878488 (+2327190us) service_pool.cc:225] Handling call
0114 02:53:59.987074 (+108586us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.068373 2302 raft_consensus.cc:695] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:00.063259 2379 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.063771 2433 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:00.039178 1581 rpcz_store.cc:269] 0114 02:53:58.260070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.774128 (+1514058us) service_pool.cc:225] Handling call
0114 02:53:59.991452 (+217324us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.056053 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 291) took 1282 ms. Trace:
I20250114 02:54:00.062078 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.076017 1660 raft_consensus.cc:1235] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: []
W20250114 02:54:00.065333 1757 debug-util.cc:398] Leaking SignalData structure 0x7b0800325280 after lost signal to thread 1760
I20250114 02:54:00.075795 1628 leader_election.cc:304] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:00.070904 2379 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.169814 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:00.077811 1757 debug-util.cc:398] Leaking SignalData structure 0x7b0800325720 after lost signal to thread 1761
I20250114 02:54:00.171671 2365 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.171994 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.075659 1657 rpcz_store.cc:276] 0114 02:53:58.773198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:58.868634 (+ 95436us) service_pool.cc:225] Handling call
0114 02:53:58.889470 (+ 20836us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:53:59.763072 (+873602us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:00.055811 (+292739us) spinlock_profiling.cc:243] Waited 847 ms on lock 0x7b54003c78d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:00.055996 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1948969856}
I20250114 02:54:00.107131 2302 consensus_queue.cc:237] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.169649 2427 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.168682 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.071920 2360 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.075451 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:54:00.170010 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 94294.6us
I20250114 02:54:00.075160 2350 raft_consensus.cc:695] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:00.170277 2387 raft_consensus.cc:2743] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:00.170528 2433 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:00.171116 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.173880 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.174731 2242 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:00.181509 1731 raft_consensus.cc:1235] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:00.190768 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:00.207003 1734 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:00.192469 2350 consensus_queue.cc:237] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.193085 2483 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.193646 1727 raft_consensus.cc:1270] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:00.190135 2392 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.194135 2368 consensus_queue.cc:1046] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:00.194800 2428 consensus_queue.cc:1035] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:00.195574 2099 consensus_queue.cc:1035] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:00.197734 1628 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:00.204593 1735 raft_consensus.cc:1270] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:00.210497 1727 raft_consensus.cc:1270] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:00.216430 1628 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
W20250114 02:54:00.204696 2484 log.cc:926] Time spent T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.378s user 0.004s sys 0.002s
I20250114 02:54:00.218700 1727 raft_consensus.cc:1270] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:00.219568 1735 raft_consensus.cc:1270] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:00.221911 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:00.222163 1628 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:00.222329 1734 raft_consensus.cc:2463] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:00.199160 1584 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
W20250114 02:54:00.224303 1585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 373) took 2222 ms (client timeout 1824 ms). Trace:
I20250114 02:54:00.211342 2242 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.224233 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.211613 2392 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.212388 2483 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:00.229344 2479 log.cc:926] Time spent T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.390s user 0.001s sys 0.001s
I20250114 02:54:00.228106 1729 raft_consensus.cc:1270] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:00.215515 2387 raft_consensus.cc:2798] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:54:00.212203 2416 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.246232 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:00.247640 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:00.249191 1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 302) took 1257 ms. Trace:
I20250114 02:54:00.283648 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:00.283987 1661 rpcz_store.cc:276] 0114 02:53:58.991828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.078675 (+ 86847us) service_pool.cc:225] Handling call
0114 02:53:59.078960 (+ 285us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:53:59.752768 (+673808us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:00.230981 (+478213us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b54003c5fd4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:00.249138 (+ 18157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1547872512}
W20250114 02:54:00.250447 1587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 389) took 1924 ms (client timeout 1888 ms). Trace:
W20250114 02:54:00.285928 1587 rpcz_store.cc:269] 0114 02:53:58.325858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.654000 (+1328142us) service_pool.cc:225] Handling call
0114 02:54:00.228162 (+574162us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b540022f954. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:00.250370 (+ 22208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":396232960}
I20250114 02:54:00.287310 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:54:00.250474 1550 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items [suppressed 20 similar messages]
W20250114 02:54:00.251096 1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 277) took 2649 ms (client timeout 2614 ms). Trace:
W20250114 02:54:00.290802 1586 rpcz_store.cc:269] 0114 02:53:57.601210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.739415 (+2138205us) service_pool.cc:225] Handling call
0114 02:54:00.231237 (+491822us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b540037b4d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:00.251023 (+ 19786us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":305846528}
I20250114 02:54:00.256554 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:00.224303 2436 raft_consensus.cc:2743] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250114 02:54:00.258929 1585 rpcz_store.cc:269] 0114 02:53:58.001759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:59.418210 (+1416451us) service_pool.cc:225] Handling call
0114 02:54:00.224126 (+805916us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b540033c2d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:00.224241 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1073193728}
I20250114 02:54:00.278111 1553 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:00.288347 1550 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34928 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 412)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 416)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 424)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 314)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 310)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 311)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 317)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 428)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 415)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 417)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 312)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 418)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 313)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 316)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 394)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 315)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 421)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 410)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 411)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 413)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 414)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 419)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 420)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 422)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 423)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 426)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 427)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 429)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 432)
I20250114 02:54:00.213194 1580 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:54:00.309615 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.285499 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
W20250114 02:54:00.300969 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 52169.6us
I20250114 02:54:00.314985 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.250556 2386 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.300163 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 388) took 1978 ms. Trace:
I20250114 02:54:00.332719 1553 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:00.298187 2346 raft_consensus.cc:2743] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:00.339663 1656 raft_consensus.cc:1235] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:00.282426 2441 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.334743 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:00.335583 1661 raft_consensus.cc:3049] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:54:00.336952 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:00.317687 1586 raft_consensus.cc:2371] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2. Re-sending same reply.
I20250114 02:54:00.342854 1660 raft_consensus.cc:3054] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:00.336491 1587 rpcz_store.cc:276] 0114 02:53:58.321877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.287026 (+1965149us) service_pool.cc:225] Handling call
0114 02:54:00.300083 (+ 13057us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.342234 1661 raft_consensus.cc:738] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:00.299605 2433 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.346303 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.345184 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.308420 2442 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.346467 1661 consensus_queue.cc:260] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.348951 1661 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:00.309471 2445 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.349032 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.349797 1587 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:00.319660 2246 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:00.321233 2387 raft_consensus.cc:695] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
W20250114 02:54:00.298892 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 76792.3us
I20250114 02:54:00.335991 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 306) took 1042 ms. Trace:
I20250114 02:54:00.346406 2441 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.361428 1661 raft_consensus.cc:1270] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:00.363672 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.331890 2248 consensus_queue.cc:1046] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20250114 02:54:00.365116 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.357754 2442 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.318326 2268 consensus_queue.cc:1035] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20250114 02:54:00.352368 2379 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:00.340590 2386 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.281862 2414 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.338342 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.417259 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:00.417987 1734 raft_consensus.cc:2388] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:00.364753 1660 raft_consensus.cc:2463] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:00.369609 1585 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:00.419052 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.419914 1585 raft_consensus.cc:2405] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:00.420558 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.370997 1587 raft_consensus.cc:2405] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:00.321764 2430 consensus_queue.cc:1035] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20250114 02:54:00.308727 2416 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.388394 2448 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.389905 1659 raft_consensus.cc:1235] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:00.392222 1729 raft_consensus.cc:1270] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:00.395588 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:00.398377 1657 raft_consensus.cc:1235] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:00.399662 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
W20250114 02:54:00.365598 2495 log.cc:926] Time spent T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.390s user 0.004s sys 0.000s
I20250114 02:54:00.321588 2106 consensus_queue.cc:1046] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20250114 02:54:00.424368 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.460323 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.419052 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:00.364051 1583 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
W20250114 02:54:00.426276 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 93648.1us
I20250114 02:54:00.427664 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:00.464475 1734 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:00.465051 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:00.465363 1734 raft_consensus.cc:3054] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:00.428980 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:00.460875 1587 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:00.430262 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:00.414657 2363 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.352932 2433 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.440349 1702 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:00.414407 2379 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:00.363296 1580 rpcz_store.cc:276] 0114 02:53:59.289516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.082837 (+793321us) service_pool.cc:225] Handling call
0114 02:54:00.331507 (+248670us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.358740 2445 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.448025 2251 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:00.457762 2357 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20250114 02:54:00.417475 2242 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.463812 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:00.467384 1736 raft_consensus.cc:2388] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:00.482090 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:00.467324 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.482764 1587 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:00.482700 1731 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:00.426239 2360 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.446802 2302 raft_consensus.cc:3049] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:00.478749 1734 raft_consensus.cc:2463] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:00.473937 1581 raft_consensus.cc:2388] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:00.473510 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 287) took 2069 ms. Trace:
I20250114 02:54:00.485987 2302 raft_consensus.cc:738] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:00.475524 2379 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:00.490823 1586 raft_consensus.cc:2363] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3.
I20250114 02:54:00.439420 2448 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.501456 1587 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:00.507591 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.480165 2508 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:00.504231 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:54:00.488099 2385 raft_consensus.cc:3049] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:54:00.484540 2511 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:00.480252 2436 consensus_queue.cc:1046] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20250114 02:54:00.519860 1587 raft_consensus.cc:2463] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:00.473554 2363 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.491350 2392 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:00.534190 2505 log.cc:926] Time spent T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250114 02:54:00.524844 2414 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.516314 2508 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.514032 2380 consensus_queue.cc:1046] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20250114 02:54:00.475625 2387 consensus_queue.cc:237] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.483789 2346 consensus_queue.cc:1035] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20250114 02:54:00.557614 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.486310 1583 rpcz_store.cc:276] 0114 02:53:58.403706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.334131 (+1930425us) service_pool.cc:225] Handling call
0114 02:54:00.469499 (+135368us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.488526 2248 consensus_queue.cc:1035] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20250114 02:54:00.516139 2385 raft_consensus.cc:738] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:00.573017 1587 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:00.572887 2380 consensus_queue.cc:1225] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer 1e5870f8269b49cf81dc39f1a5c8f5c1 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 02:54:00.509167 2483 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.549023 2360 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.569254 2513 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:00.529219 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
W20250114 02:54:00.600685 2506 log.cc:926] Time spent T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
I20250114 02:54:00.600935 1580 raft_consensus.cc:2371] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2. Re-sending same reply.
I20250114 02:54:00.551990 1578 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:00.515902 2430 consensus_queue.cc:1035] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20250114 02:54:00.585374 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 286) took 2182 ms. Trace:
I20250114 02:54:00.585857 1587 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:00.627209 1586 rpcz_store.cc:276] 0114 02:53:58.403456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.343106 (+1939650us) service_pool.cc:225] Handling call
0114 02:54:00.399482 (+ 56376us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:54:00.585210 (+185728us) spinlock_profiling.cc:243] Waited 48.6 ms on lock 0x7b54000ca8d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:00.585323 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116789504}
I20250114 02:54:00.627722 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.577510 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.628278 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.628809 1586 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:00.629330 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 301) took 1357 ms. Trace:
I20250114 02:54:00.629701 1586 rpcz_store.cc:276] 0114 02:53:59.271924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.628090 (+1356166us) service_pool.cc:225] Handling call
0114 02:54:00.629266 (+ 1176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.549794 2511 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.596093 1582 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:00.578727 2448 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:00.599253 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.600023 2433 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:00.601970 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.604231 2416 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:00.600842 2513 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.606752 1578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 303) took 1329 ms. Trace:
I20250114 02:54:00.608428 2251 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.609475 2363 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:00.610069 2445 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:00.584257 2441 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:00.633396 2448 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.612557 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.618415 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:00.619040 2385 consensus_queue.cc:260] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.654815 2385 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:00.657867 1579 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:00.619735 2437 raft_consensus.cc:3049] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:00.659150 2437 raft_consensus.cc:738] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:00.660135 2437 consensus_queue.cc:260] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.661576 1580 raft_consensus.cc:2388] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:00.617686 2302 consensus_queue.cc:260] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.621316 2386 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.638728 1578 rpcz_store.cc:276] 0114 02:53:59.277198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.171731 (+894533us) service_pool.cc:225] Handling call
0114 02:54:00.606695 (+434964us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.663444 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.664036 1578 raft_consensus.cc:3054] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:00.667476 2302 raft_consensus.cc:3054] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:00.643460 1581 raft_consensus.cc:2388] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
W20250114 02:54:00.632834 1582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 367) took 2739 ms (client timeout 2384 ms). Trace:
I20250114 02:54:00.632541 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.571913 2512 consensus_queue.cc:1046] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20250114 02:54:00.599339 2508 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:00.639470 1587 raft_consensus.cc:2463] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:00.586776 2442 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:00.640334 2246 raft_consensus.cc:2743] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:00.662124 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 304) took 1381 ms. Trace:
I20250114 02:54:00.661583 2437 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:00.682489 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.683424 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 394) took 1281 ms. Trace:
I20250114 02:54:00.683907 1586 rpcz_store.cc:276] 0114 02:53:59.402313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.682219 (+1279906us) service_pool.cc:225] Handling call
0114 02:54:00.683358 (+ 1139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.684897 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:00.642812 2416 raft_consensus.cc:2798] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:54:00.621068 2379 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.635259 2433 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.611773 2360 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:00.672816 1582 rpcz_store.cc:269] 0114 02:53:57.893708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.223984 (+2330276us) service_pool.cc:225] Handling call
0114 02:54:00.632771 (+408787us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.678789 2512 raft_consensus.cc:3049] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:54:00.680115 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.645156 2363 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.686584 1483 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39732 (request call id 91) took 7033 ms. Trace:
I20250114 02:54:00.685545 1586 raft_consensus.cc:2463] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:00.729557 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 409) took 1175 ms. Trace:
I20250114 02:54:00.729960 1586 rpcz_store.cc:276] 0114 02:53:59.554060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.684609 (+1130549us) service_pool.cc:225] Handling call
0114 02:54:00.729494 (+ 44885us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.730995 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.731575 1586 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:00.702526 1587 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:00.680619 1580 rpcz_store.cc:276] 0114 02:53:59.280514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.601675 (+1321161us) service_pool.cc:225] Handling call
0114 02:54:00.662060 (+ 60385us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.714774 2433 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.680124 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.717125 2360 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.718000 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:00.718499 2512 raft_consensus.cc:738] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:00.714054 1578 raft_consensus.cc:2463] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:00.723836 1585 raft_consensus.cc:2388] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:00.724790 2379 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.727969 2511 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.671547 2448 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.756937 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 310) took 1007 ms. Trace:
I20250114 02:54:00.757365 1585 rpcz_store.cc:276] 0114 02:53:59.749273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.420366 (+671093us) service_pool.cc:225] Handling call
0114 02:54:00.756718 (+336352us) spinlock_profiling.cc:243] Waited 302 ms on lock 0x7b54002120d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:00.756873 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":695529344}
I20250114 02:54:00.759454 1585 raft_consensus.cc:1270] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:00.760195 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 345) took 3702 ms. Trace:
I20250114 02:54:00.760646 1585 rpcz_store.cc:276] 0114 02:53:57.058360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.758298 (+3699938us) service_pool.cc:225] Handling call
0114 02:54:00.759062 (+ 764us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:00.759920 (+ 858us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:00.760138 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.762352 1585 raft_consensus.cc:1235] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:00.763185 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 346) took 3701 ms. Trace:
I20250114 02:54:00.763629 1585 rpcz_store.cc:276] 0114 02:53:57.061404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.761839 (+3700435us) service_pool.cc:225] Handling call
0114 02:54:00.762155 (+ 316us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:00.762909 (+ 754us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:00.763128 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.702173 2513 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.743005 1587 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:00.768855 1585 raft_consensus.cc:1235] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:54:00.751958 1586 raft_consensus.cc:2463] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:00.693596 2416 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:00.735433 1483 rpcz_store.cc:276] 0114 02:53:53.643481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:53.643593 (+ 112us) service_pool.cc:225] Handling call
0114 02:53:54.020545 (+376952us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54002b02d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605ad50 00007fbce60ddf2a 00007fbcebdf9dec 00007fbcebef1dee 00007fbcea444fc4 00007fbcea444f52 00007fbcea444e81 00007fbcea444dfc 00007fbcea4440b2 00007fbce4500dfc 00007fbce4500236
0114 02:54:00.677193 (+6656648us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:53:59.455787 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:53:59.466111 (+ 10324us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:53:59.466144 (+ 33us) write_op.cc:435] Acquired schema lock
0114 02:53:59.466164 (+ 20us) tablet.cc:659] Decoding operations
0114 02:53:59.649672 (+183508us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:53:59.661939 (+ 12267us) write_op.cc:641] Partition lock acquired for write op
0114 02:53:59.682387 (+ 20448us) tablet.cc:682] Acquiring locks for 6 operations
0114 02:53:59.774457 (+ 92070us) tablet.cc:698] Row locks acquired
0114 02:53:59.775942 (+ 1485us) write_op.cc:260] PREPARE: finished
0114 02:53:59.826659 (+ 50717us) write_op.cc:270] Start()
0114 02:53:59.829845 (+ 3186us) write_op.cc:276] Timestamp: P: 1736823239823446 usec, L: 0
0114 02:53:59.833186 (+ 3341us) op_driver.cc:348] REPLICATION: starting
0114 02:53:59.918670 (+ 85484us) log.cc:844] Serialized 2463 byte log entry
0114 02:54:00.339695 (+421025us) op_driver.cc:464] REPLICATION: finished
0114 02:54:00.505825 (+166130us) write_op.cc:301] APPLY: starting
0114 02:54:00.509609 (+ 3784us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:00.569358 (+ 59749us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:00.591661 (+ 22303us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:00.612590 (+ 20929us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:00.612800 (+ 210us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0114 02:54:00.612850 (+ 50us) write_op.cc:312] APPLY: finished
0114 02:54:00.613103 (+ 253us) log.cc:844] Serialized 52 byte log entry
0114 02:54:00.613273 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:00.613375 (+ 102us) write_op.cc:454] Released schema lock
0114 02:54:00.613419 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:00.613740 (+ 321us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:00.613776 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1352621,"reported_tablets":8,"spinlock_wait_cycles":381041920,"child_traces":[["op",{"apply.queue_time_us":140586,"apply.run_cpu_time_us":3057,"apply.run_wall_time_us":124366,"num_ops":6,"prepare.queue_time_us":42165,"prepare.run_cpu_time_us":35240,"prepare.run_wall_time_us":601928,"raft.queue_time_us":160179,"raft.run_cpu_time_us":11954,"raft.run_wall_time_us":42562,"replication_time_us":506364,"thread_start_us":134884,"threads_started":4,"wal-append.queue_time_us":208533}]]}
I20250114 02:54:00.753901 1581 raft_consensus.cc:2371] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2. Re-sending same reply.
I20250114 02:54:00.775451 1587 raft_consensus.cc:1270] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:00.728164 1482 catalog_manager.cc:5526] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 1 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:00.730021 2363 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.774533 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 347) took 3710 ms. Trace:
I20250114 02:54:00.794016 1585 rpcz_store.cc:276] 0114 02:53:57.064386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.764587 (+3700201us) service_pool.cc:225] Handling call
0114 02:54:00.768623 (+ 4036us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:00.769373 (+ 750us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:00.774315 (+ 4942us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b540016ce50. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d4a09 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:00.774469 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8435584}
I20250114 02:54:00.785821 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 348) took 3616 ms. Trace:
I20250114 02:54:00.778685 2416 raft_consensus.cc:3054] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:00.775944 1582 raft_consensus.cc:2388] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:00.796097 1587 rpcz_store.cc:276] 0114 02:53:57.169114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.774395 (+3605281us) service_pool.cc:225] Handling call
0114 02:54:00.775102 (+ 707us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:00.785575 (+ 10473us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:00.785764 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.773494 1579 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:00.816761 1587 raft_consensus.cc:1235] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:00.818215 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 251) took 3273 ms. Trace:
W20250114 02:54:00.827232 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 386981us
I20250114 02:54:00.820516 1587 rpcz_store.cc:276] 0114 02:53:57.545159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.815661 (+3270502us) service_pool.cc:225] Handling call
0114 02:54:00.816545 (+ 884us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:00.818005 (+ 1460us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:00.818165 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.829890 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:00.832949 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:00.820367 1579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 378) took 2636 ms (client timeout 3229 ms). Trace:
I20250114 02:54:00.833931 1587 raft_consensus.cc:1270] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:00.821003 1584 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:00.834293 1702 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:00.838088 1702 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:00.836174 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 413) took 1041 ms. Trace:
I20250114 02:54:00.840094 1587 rpcz_store.cc:276] 0114 02:53:59.794547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.833427 (+1038880us) service_pool.cc:225] Handling call
0114 02:54:00.833666 (+ 239us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:00.835921 (+ 2255us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:00.836118 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:00.830938 2430 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.830288 2448 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:00.845063 1702 leader_election.cc:400] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3.
I20250114 02:54:00.843279 1587 raft_consensus.cc:1270] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 02:54:00.785334 2512 consensus_queue.cc:260] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.832571 2433 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:00.845420 1702 leader_election.cc:403] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250114 02:54:00.847301 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:00.852355 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:00.853672 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:00.848248 2430 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.857115 2430 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:00.848706 2442 raft_consensus.cc:2798] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:54:00.859068 2442 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:00.859493 2442 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:54:00.835726 1579 rpcz_store.cc:269] 0114 02:53:58.184354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.228965 (+2044611us) service_pool.cc:225] Handling call
0114 02:54:00.612421 (+383456us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315ed0e 0000000000353db9 00007fbceafbea50 00007fbceb084814 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:54:00.819071 (+206650us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4950400}
I20250114 02:54:00.836658 2302 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.840744 2441 raft_consensus.cc:2743] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:00.835204 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:00.846378 1587 pending_rounds.cc:77] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 1
I20250114 02:54:00.978201 1587 pending_rounds.cc:99] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 2.2
W20250114 02:54:00.923906 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 488398us
I20250114 02:54:00.866667 1582 raft_consensus.cc:1235] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:00.883652 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:00.986085 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 141192us
I20250114 02:54:00.962859 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:00.880358 1656 raft_consensus.cc:1270] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:00.879547 2385 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.931648 2363 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:00.871711 2302 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.943827 2529 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.978477 1587 raft_consensus.cc:2883] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:00.870797 1579 raft_consensus.cc:1270] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250114 02:54:00.919351 2526 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.019959 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 351) took 3840 ms. Trace:
I20250114 02:54:00.885164 1581 raft_consensus.cc:1270] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 02:54:01.013847 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 414) took 1109 ms. Trace:
I20250114 02:54:00.880158 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 302) took 1606 ms. Trace:
I20250114 02:54:00.945959 2527 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.885483 2512 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:01.009954 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.014461 1628 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:00.928197 2522 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.987725 2530 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.027772 1586 rpcz_store.cc:276] 0114 02:53:57.179193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.777285 (+3598092us) service_pool.cc:225] Handling call
0114 02:54:00.777574 (+ 289us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:00.777781 (+ 207us) raft_consensus.cc:1529] Early marking committed up to index 2
0114 02:54:00.940889 (+163108us) log.cc:844] Serialized 14 byte log entry
0114 02:54:01.019602 (+ 78713us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:01.019665 (+ 63us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:01.019715 (+ 50us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.019777 (+ 62us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:01.019899 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13785,"prepare.queue_time_us":201820,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":8466,"raft.queue_time_us":6481,"thread_start_us":218282,"threads_started":2}
I20250114 02:54:01.042122 1586 raft_consensus.cc:1270] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 02:54:01.042587 1586 pending_rounds.cc:77] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 1
I20250114 02:54:01.042886 1586 pending_rounds.cc:99] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 02:54:01.043146 1586 raft_consensus.cc:2883] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:01.031715 1587 rpcz_store.cc:276] 0114 02:53:59.904602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.842241 (+937639us) service_pool.cc:225] Handling call
0114 02:54:00.842874 (+ 633us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:01.013275 (+170401us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.013722 (+ 447us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:01.015406 1579 pending_rounds.cc:77] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 0
I20250114 02:54:01.013531 1654 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.013367 2519 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.007045 2242 consensus_queue.cc:1035] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.025s
I20250114 02:54:01.005640 2387 consensus_queue.cc:1035] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20250114 02:54:01.005995 2385 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:00.999827 2531 raft_consensus.cc:2743] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3.
I20250114 02:54:00.931237 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.005856 2513 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.986794 1702 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:01.015472 2302 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.031622 1581 pending_rounds.cc:77] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 1
I20250114 02:54:01.055805 1581 pending_rounds.cc:99] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 02:54:01.009217 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 411) took 1269 ms. Trace:
I20250114 02:54:01.021404 2511 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.010879 2197 consensus_queue.cc:1035] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20250114 02:54:01.028266 2379 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.045518 1587 raft_consensus.cc:1270] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 02:54:01.059841 1587 pending_rounds.cc:77] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 1
I20250114 02:54:01.060132 1587 pending_rounds.cc:99] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 02:54:01.061383 2379 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.063536 2379 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.059362 1582 rpcz_store.cc:276] 0114 02:53:59.739904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.828141 (+1088237us) service_pool.cc:225] Handling call
0114 02:54:00.837716 (+ 9575us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:00.985590 (+147874us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.009163 (+ 23573us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:01.064256 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:01.064757 1582 raft_consensus.cc:1235] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:01.065006 1659 raft_consensus.cc:2388] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:01.065353 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 423) took 1083 ms. Trace:
I20250114 02:54:01.060415 1587 raft_consensus.cc:2883] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:01.065691 1582 rpcz_store.cc:276] 0114 02:53:59.981789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:01.064308 (+1082519us) service_pool.cc:225] Handling call
0114 02:54:01.064600 (+ 292us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:01.065128 (+ 528us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.065302 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:01.066054 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 422) took 1121 ms. Trace:
I20250114 02:54:01.066380 1587 rpcz_store.cc:276] 0114 02:53:59.944833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:01.044985 (+1100152us) service_pool.cc:225] Handling call
0114 02:54:01.045221 (+ 236us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:01.065849 (+ 20628us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.066010 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:01.068058 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.079628 1587 raft_consensus.cc:1270] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:01.015619 2368 consensus_queue.cc:1035] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20250114 02:54:01.044359 1579 pending_rounds.cc:99] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250114 02:54:01.029757 2520 consensus_queue.cc:1035] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20250114 02:54:01.053655 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 352) took 3874 ms. Trace:
W20250114 02:54:01.057621 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 70907.1us
I20250114 02:54:01.043870 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 420) took 1134 ms. Trace:
I20250114 02:54:01.046224 1656 raft_consensus.cc:2463] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:01.097708 1586 rpcz_store.cc:276] 0114 02:53:59.909914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:01.041321 (+1131407us) service_pool.cc:225] Handling call
0114 02:54:01.041695 (+ 374us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:01.043602 (+ 1907us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.043804 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:01.081970 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 427) took 1015 ms. Trace:
I20250114 02:54:01.100678 1587 rpcz_store.cc:276] 0114 02:54:00.066644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:01.067007 (+1000363us) service_pool.cc:225] Handling call
0114 02:54:01.067229 (+ 222us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:01.081693 (+ 14464us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.081913 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:01.095419 1579 raft_consensus.cc:2883] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:01.095698 1585 rpcz_store.cc:276] 0114 02:53:57.179491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.795256 (+3615765us) service_pool.cc:225] Handling call
0114 02:54:00.795544 (+ 288us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:00.795743 (+ 199us) raft_consensus.cc:1529] Early marking committed up to index 1
0114 02:54:01.014348 (+218605us) log.cc:844] Serialized 14 byte log entry
0114 02:54:01.053185 (+ 38837us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:01.053268 (+ 83us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:01.053338 (+ 70us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.053461 (+ 123us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:01.053604 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1080,"prepare.queue_time_us":234150,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":826,"raft.queue_time_us":1252,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":2087,"thread_start_us":235544,"threads_started":2}
I20250114 02:54:01.047510 2521 raft_consensus.cc:2798] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:01.103184 1585 raft_consensus.cc:1270] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 02:54:01.103600 1585 pending_rounds.cc:77] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 1
I20250114 02:54:01.103828 1585 pending_rounds.cc:99] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 02:54:01.104038 1585 raft_consensus.cc:2883] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:01.056102 1581 raft_consensus.cc:2883] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:01.105414 1585 raft_consensus.cc:1270] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:01.100131 1586 raft_consensus.cc:1270] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:01.059841 2531 raft_consensus.cc:2798] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:01.048457 2428 consensus_queue.cc:1035] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20250114 02:54:01.047396 2229 consensus_queue.cc:1035] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20250114 02:54:01.106647 1587 raft_consensus.cc:1235] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:01.113466 1553 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:01.107328 1702 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:01.117646 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 429) took 1050 ms. Trace:
I20250114 02:54:01.129069 1586 rpcz_store.cc:276] 0114 02:54:00.071072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:01.098747 (+1027675us) service_pool.cc:225] Handling call
0114 02:54:01.099093 (+ 346us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:01.113487 (+ 14394us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.117574 (+ 4087us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:01.034979 1584 rpcz_store.cc:276] 0114 02:53:59.274007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.335253 (+1061246us) service_pool.cc:225] Handling call
0114 02:54:00.879966 (+544713us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54001aa9d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:00.880098 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":707795712}
I20250114 02:54:01.115563 1585 raft_consensus.cc:1270] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:01.128290 1702 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:01.136752 1702 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:01.138063 1702 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
W20250114 02:54:01.139000 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 90216.7us
I20250114 02:54:01.150434 1702 leader_election.cc:400] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3.
I20250114 02:54:01.150689 1702 leader_election.cc:403] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250114 02:54:01.054375 2513 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.111446 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 419) took 1201 ms. Trace:
I20250114 02:54:01.112555 1578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 349) took 3943 ms. Trace:
I20250114 02:54:01.048204 2519 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.045380 2522 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.031175 2526 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.162413 1702 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:01.116178 2363 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.046051 2530 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.134977 2246 raft_consensus.cc:2798] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:01.123344 2531 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:01.164790 1578 rpcz_store.cc:276] 0114 02:53:57.169390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.774285 (+3604895us) service_pool.cc:225] Handling call
0114 02:54:00.778458 (+ 4173us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:00.803363 (+ 24905us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:00.817202 (+ 13839us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:00.836530 (+ 19328us) log.cc:844] Serialized 26 byte log entry
0114 02:54:00.984418 (+147888us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:01.029617 (+ 45199us) log.cc:844] Serialized 14 byte log entry
0114 02:54:01.038267 (+ 8650us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.040994 (+ 2727us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:01.073779 (+ 32785us) raft_consensus.cc:1680] finished
0114 02:54:01.096958 (+ 23179us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:01.107905 (+ 10947us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2932,"prepare.queue_time_us":56209,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":6848,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":16166,"thread_start_us":105408,"threads_started":1,"wal-append.queue_time_us":171788}
I20250114 02:54:01.175205 2531 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.175602 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.124013 2521 raft_consensus.cc:695] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:01.175843 2363 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.177737 2363 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.177158 2521 consensus_queue.cc:237] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.178242 2363 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:01.038465 2527 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.178613 2363 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:01.179040 2363 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.180198 2527 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.162498 2433 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:54:01.181205 2433 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:01.181699 2433 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:01.021970 2529 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.161677 1579 rpcz_store.cc:276] 0114 02:53:59.909619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.863911 (+954292us) service_pool.cc:225] Handling call
0114 02:54:00.864267 (+ 356us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:01.101785 (+237518us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.111389 (+ 9604us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:01.159795 2441 raft_consensus.cc:2798] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:01.173161 2513 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.173311 2530 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.128851 2528 raft_consensus.cc:2743] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:01.148869 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.190431 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 342) took 4168 ms. Trace:
I20250114 02:54:01.190850 1580 rpcz_store.cc:276] 0114 02:53:57.022435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.754874 (+3732439us) service_pool.cc:225] Handling call
0114 02:54:00.766892 (+ 12018us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:00.767089 (+ 197us) raft_consensus.cc:1529] Early marking committed up to index 2
0114 02:54:01.036157 (+269068us) log.cc:844] Serialized 14 byte log entry
0114 02:54:01.169385 (+133228us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:01.186614 (+ 17229us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:01.190110 (+ 3496us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.190215 (+ 105us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:01.190369 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":195732,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":7586,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":5130,"thread_start_us":368495,"threads_started":2}
I20250114 02:54:01.193646 2529 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.164664 2508 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.197014 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.197751 1661 raft_consensus.cc:2463] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
W20250114 02:54:01.167130 1583 consensus_meta.cc:287] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.525s user 0.010s sys 0.000s
I20250114 02:54:01.199555 1553 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:01.096212 2511 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.163949 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 410) took 1593 ms. Trace:
I20250114 02:54:01.166509 2523 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.206952 2385 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.208730 2522 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.188959 2441 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:01.203052 1655 raft_consensus.cc:2463] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:01.170120 2246 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:01.202342 2302 raft_consensus.cc:2798] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250114 02:54:01.220973 2540 log.cc:926] Time spent T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20250114 02:54:01.201334 1583 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:01.232255 2519 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.210515 2526 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.193990 2363 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.224498 2441 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.224499 2246 raft_consensus.cc:3054] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.228272 2302 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:01.247040 2512 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:01.246764 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 307) took 1957 ms. Trace:
I20250114 02:54:01.226908 2416 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.261171 1582 raft_consensus.cc:1270] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 02:54:01.262046 2528 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.263257 2302 raft_consensus.cc:3054] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:01.216195 1581 rpcz_store.cc:276] 0114 02:53:59.570107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.817324 (+1247217us) service_pool.cc:225] Handling call
0114 02:54:00.884617 (+ 67293us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:54:00.884799 (+ 182us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:01.124349 (+239550us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.160843 (+ 36494us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3363200}
I20250114 02:54:01.275141 1582 pending_rounds.cc:77] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 1
I20250114 02:54:01.273136 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.271378 1583 rpcz_store.cc:276] 0114 02:53:59.289778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.584746 (+1294968us) service_pool.cc:225] Handling call
0114 02:54:01.245063 (+660317us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:01.283228 1582 pending_rounds.cc:99] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 02:54:01.273849 2512 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.294350 1582 raft_consensus.cc:2883] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250114 02:54:01.300554 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 138223us
I20250114 02:54:01.305922 1702 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:01.296962 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.280212 2528 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.307524 1702 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:01.283589 2442 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.310317 2433 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.317114 2433 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.308743 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.319164 1656 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:01.320333 2363 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.453358 2242 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:01.417949 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 255009us
I20250114 02:54:01.323526 1553 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
W20250114 02:54:01.453145 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 143441us
I20250114 02:54:01.304072 2511 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.352182 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 426) took 1285 ms. Trace:
I20250114 02:54:01.323127 2530 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:01.459633 1582 rpcz_store.cc:276] 0114 02:54:00.066302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:01.066405 (+1000103us) service_pool.cc:225] Handling call
0114 02:54:01.116402 (+ 49997us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:01.306958 (+190556us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.322956 (+ 15998us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b5400291654. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:01.331833 (+ 8877us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":241429376}
I20250114 02:54:01.441157 2430 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.458106 2360 raft_consensus.cc:2798] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:01.457206 1702 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:01.461393 2360 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:01.461802 2360 raft_consensus.cc:3054] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.462253 2549 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.458492 2448 raft_consensus.cc:491] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:01.456487 2242 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:01.456082 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 132674us
I20250114 02:54:01.466305 2448 raft_consensus.cc:513] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.467334 2550 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.467458 2242 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.384526 2246 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.467835 2550 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.468593 2448 leader_election.cc:290] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.469394 2246 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.458799 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 326) took 1026 ms. Trace:
I20250114 02:54:01.470060 2550 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.320809 2416 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.321631 2523 raft_consensus.cc:2743] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:01.438731 2302 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.323596 2529 raft_consensus.cc:2798] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:01.471714 2430 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.472245 2529 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:01.472788 2529 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.473374 2302 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.473495 2430 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.410588 2441 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.475487 2441 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.396257 2531 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.457499 1628 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:01.471052 1553 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:01.477669 2521 raft_consensus.cc:2798] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:54:01.477739 2531 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.478125 2521 raft_consensus.cc:695] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:01.453640 2442 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.320947 1654 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:01.478965 2521 consensus_queue.cc:237] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.388322 2512 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625)
I20250114 02:54:01.480755 2430 raft_consensus.cc:2798] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:01.399371 2508 raft_consensus.cc:2743] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3.
I20250114 02:54:01.483203 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:01.483187 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.483927 1658 raft_consensus.cc:3054] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.483772 2529 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.484340 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:01.484871 1655 raft_consensus.cc:3054] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:01.484998 2508 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.485594 2529 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.485529 2508 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.483342 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.387634 2528 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.470054 1702 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:01.486903 2530 raft_consensus.cc:2743] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:01.487723 2430 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:01.489818 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.492388 1655 raft_consensus.cc:2463] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:01.480957 2360 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.493060 1658 raft_consensus.cc:2463] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:01.489859 2430 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.499106 1553 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:01.494439 1628 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:01.501065 1702 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:01.502379 1702 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:01.502911 1628 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:01.503625 1628 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:01.509621 1628 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:01.512205 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.512351 1553 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:01.513012 1579 raft_consensus.cc:2388] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:01.513070 2246 raft_consensus.cc:2798] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:01.513865 2246 raft_consensus.cc:695] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:01.527364 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.527442 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.528064 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.528833 1580 raft_consensus.cc:2363] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3.
I20250114 02:54:01.528703 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:01.528993 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.529487 1656 raft_consensus.cc:2388] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:01.529726 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.505232 2302 raft_consensus.cc:2798] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:01.530365 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:01.530689 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.505373 2530 consensus_queue.cc:1046] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20250114 02:54:01.531563 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.531733 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.532735 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.532922 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.533726 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.520630 1628 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:01.535730 1628 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:01.535909 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.537127 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.537698 1628 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:01.524457 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.538667 1587 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:01.539458 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.539999 1587 raft_consensus.cc:2463] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:01.540462 2508 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.540747 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.541848 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.523715 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.543139 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.543277 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.543756 1655 raft_consensus.cc:2463] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:01.523020 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.544252 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:01.527675 1658 raft_consensus.cc:1270] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:01.526217 1702 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:01.530793 2302 raft_consensus.cc:695] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:01.525312 1586 raft_consensus.cc:1235] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:01.514621 1553 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:01.528286 1654 raft_consensus.cc:2388] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:01.526036 1585 raft_consensus.cc:1235] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:01.547010 1553 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:01.547122 1585 raft_consensus.cc:1235] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:01.542377 1587 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:01.537747 1653 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:01.547993 1553 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:01.549887 2106 raft_consensus.cc:2798] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:01.534171 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.548090 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.551659 1586 raft_consensus.cc:1235] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:01.552361 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:01.554430 1585 raft_consensus.cc:2388] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:01.554450 1587 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:01.550024 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.506371 2436 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:01.553025 2511 raft_consensus.cc:491] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.548090 2441 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.556272 2511 raft_consensus.cc:513] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.527714 2246 consensus_queue.cc:237] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.557020 2441 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.558154 2511 leader_election.cc:290] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.549151 2442 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:01.558621 2437 consensus_queue.cc:1046] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:01.559056 2441 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.548817 2531 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.550558 2521 raft_consensus.cc:2743] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:01.530287 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:01.548245 2526 raft_consensus.cc:2798] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:54:01.559670 2531 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.514461 2529 raft_consensus.cc:2798] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:01.556500 1582 raft_consensus.cc:1270] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:01.562734 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.564568 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.564736 1585 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 1.
I20250114 02:54:01.560921 1702 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:01.561103 1628 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:01.514371 2512 raft_consensus.cc:2798] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:01.548538 2099 raft_consensus.cc:2798] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:54:01.563051 1553 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:01.563314 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:01.573832 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.574438 1585 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:01.574517 1553 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:01.559527 2523 raft_consensus.cc:2798] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:01.545796 2302 consensus_queue.cc:237] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.506500 2518 raft_consensus.cc:2798] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:54:01.574478 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:01.499109 2528 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:01.556416 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.555665 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.563894 2242 raft_consensus.cc:2743] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:01.575554 1702 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:01.566144 2522 raft_consensus.cc:2743] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:01.561893 2428 raft_consensus.cc:2798] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:01.587870 1553 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:01.607111 1586 raft_consensus.cc:2388] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:01.608456 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:01.530308 1580 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:01.572427 2106 raft_consensus.cc:695] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:01.610816 1702 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:01.573072 2526 raft_consensus.cc:695] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:01.571602 2508 raft_consensus.cc:3049] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:54:01.577317 2441 raft_consensus.cc:2743] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:54:01.577006 2520 raft_consensus.cc:2798] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
W20250114 02:54:01.684265 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 73520.4us
I20250114 02:54:01.607949 2099 raft_consensus.cc:695] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:01.565898 2530 consensus_queue.cc:1046] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.117s
I20250114 02:54:01.680562 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.611860 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.711252 1586 raft_consensus.cc:2463] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:01.711402 1585 raft_consensus.cc:2463] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:01.712563 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.712755 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:01.713373 1585 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:01.579131 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:01.616945 2523 raft_consensus.cc:695] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:01.559123 2442 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.576910 2385 raft_consensus.cc:2743] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:01.709025 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:01.566032 2524 raft_consensus.cc:2798] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:54:01.579239 2513 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:01.578768 2512 raft_consensus.cc:695] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:01.652604 2363 raft_consensus.cc:2798] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:01.706530 2508 raft_consensus.cc:738] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:01.572415 2529 raft_consensus.cc:695] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:01.677469 2483 consensus_queue.cc:1035] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20250114 02:54:01.609483 2521 consensus_queue.cc:1035] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.118s
I20250114 02:54:01.653684 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.575726 2229 consensus_queue.cc:1035] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.131s
I20250114 02:54:01.678427 2528 raft_consensus.cc:695] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:01.683907 2246 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.684201 2527 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.722815 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:01.743008 1736 raft_consensus.cc:2405] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for term 3 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:01.724933 1735 raft_consensus.cc:2388] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:01.609449 2360 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.709833 2242 raft_consensus.cc:2743] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:01.710264 2433 raft_consensus.cc:2798] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:01.565747 2543 consensus_queue.cc:1035] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.157s
I20250114 02:54:01.677286 2416 consensus_queue.cc:1035] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20250114 02:54:01.711995 2448 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:01.676633 2346 consensus_queue.cc:1046] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20250114 02:54:01.684485 2387 consensus_queue.cc:1046] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20250114 02:54:01.739467 2246 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.739432 2099 consensus_queue.cc:237] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.726858 2513 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:01.728787 2363 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:01.733865 2524 raft_consensus.cc:695] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:01.723786 2368 consensus_queue.cc:1035] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250114 02:54:01.748514 2433 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:01.753777 2448 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.807426 2433 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:01.784453 2523 consensus_queue.cc:237] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.758177 2520 raft_consensus.cc:695] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:01.812299 2436 raft_consensus.cc:2743] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:01.783936 2513 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.816481 2549 raft_consensus.cc:2743] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:01.807200 2442 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.755105 2526 consensus_queue.cc:237] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.798238 2529 consensus_queue.cc:237] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.747712 2527 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.756570 2531 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.819211 2527 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.685308 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.846591 1579 raft_consensus.cc:2405] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250114 02:54:01.731575 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 79228us
I20250114 02:54:01.849238 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:54:01.609143 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:54:01.710389 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:01.683663 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:54:01.694856 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 107055us
I20250114 02:54:01.853638 1584 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:01.651705 1583 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:01.723124 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:01.855526 1586 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.856442 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:01.607520 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.714753 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.861049 1585 raft_consensus.cc:2463] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:01.747334 1733 pending_rounds.cc:77] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 1
I20250114 02:54:01.865231 1733 pending_rounds.cc:99] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 02:54:01.865965 1733 raft_consensus.cc:2883] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:01.870925 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:01.872146 1734 raft_consensus.cc:2388] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:01.470234 1729 rpcz_store.cc:276] 0114 02:54:00.432608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:00.480799 (+ 48191us) service_pool.cc:225] Handling call
0114 02:54:00.515916 (+ 35117us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:00.930425 (+414509us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:00.943748 (+ 13323us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:01.014256 (+ 70508us) log.cc:844] Serialized 26 byte log entry
0114 02:54:01.299244 (+284988us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:01.319691 (+ 20447us) log.cc:844] Serialized 14 byte log entry
0114 02:54:01.454585 (+134894us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:01.458467 (+ 3882us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:01.458521 (+ 54us) raft_consensus.cc:1680] finished
0114 02:54:01.458559 (+ 38us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:01.458737 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":18090,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":61,"raft.queue_time_us":75431,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":211,"thread_start_us":602353,"threads_started":2,"wal-append.queue_time_us":275180}
I20250114 02:54:01.873574 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:01.875146 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:01.875989 1736 raft_consensus.cc:2463] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:01.877435 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:01.877561 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:01.814435 2430 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.817271 2433 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.808842 2448 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.758846 2508 consensus_queue.cc:260] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.808590 2512 consensus_queue.cc:237] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.902971 2433 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.812039 2520 consensus_queue.cc:237] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.807680 2524 consensus_queue.cc:237] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.800253 2528 consensus_queue.cc:237] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.790832 2518 raft_consensus.cc:695] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
W20250114 02:54:01.806481 2556 log.cc:926] Time spent T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20250114 02:54:01.878252 1736 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:01.884995 2387 raft_consensus.cc:3049] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:01.886168 2246 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.886772 2430 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:01.887007 2531 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:01.887888 1587 raft_consensus.cc:2388] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:01.874543 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:01.750806 2428 raft_consensus.cc:695] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:01.887842 2368 raft_consensus.cc:3049] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:01.908149 2518 consensus_queue.cc:237] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.940285 2428 consensus_queue.cc:237] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.941190 1729 raft_consensus.cc:2463] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:01.942641 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:01.745955 2106 consensus_queue.cc:237] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.938109 2531 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.783593 2363 raft_consensus.cc:3054] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.945822 2531 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:01.936172 2368 raft_consensus.cc:738] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:01.952634 2368 consensus_queue.cc:260] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:01.954102 2368 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.908505 2387 raft_consensus.cc:738] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:01.960225 1732 raft_consensus.cc:1270] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:01.961891 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:01.966037 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:01.966940 1736 raft_consensus.cc:2463] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:01.967988 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:01.968752 1736 raft_consensus.cc:2463] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:01.969650 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:01.971690 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:01.972692 1728 raft_consensus.cc:2463] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:01.972985 2562 consensus_queue.cc:1035] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:01.973845 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:01.975327 1736 raft_consensus.cc:2405] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250114 02:54:01.974649 2229 consensus_queue.cc:1035] T ee328440ef8f409ca059ac947d40a58a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20250114 02:54:01.984509 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:01.988266 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:01.989295 1654 raft_consensus.cc:3054] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:01.990872 2561 consensus_queue.cc:1035] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:01.984359 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.003720 1586 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:01.993239 2346 consensus_queue.cc:1035] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20250114 02:54:02.007129 2560 consensus_queue.cc:1035] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:02.016285 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:02.016839 2229 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:02.002415 2387 consensus_queue.cc:260] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.019016 2387 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.024359 2558 consensus_queue.cc:1035] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:02.026928 2099 consensus_queue.cc:1035] T 464b2086919241de965a1ac17809451c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:02.027432 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:02.029294 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:02.029783 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:02.030174 1732 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:02.030748 1727 raft_consensus.cc:2463] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:02.031349 2197 raft_consensus.cc:3049] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:54:02.031740 2197 raft_consensus.cc:738] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:02.032068 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:02.032974 2197 consensus_queue.cc:260] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.033480 1727 raft_consensus.cc:2405] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250114 02:54:02.043490 2568 log.cc:926] Time spent T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20250114 02:54:02.044860 2559 consensus_queue.cc:1035] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:02.046056 1735 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:54:02.046887 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:02.047817 1730 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:02.017750 2229 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.047051 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:02.050047 1735 raft_consensus.cc:3054] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:02.051837 2363 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.038166 1734 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:02.055573 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:02.031302 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:02.059240 1732 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:02.059396 2526 consensus_queue.cc:1035] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:02.059756 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:02.060379 1731 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.063107 2197 raft_consensus.cc:3054] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:02.054409 2363 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:02.068800 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:02.063158 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.077400 1654 raft_consensus.cc:2463] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:02.080842 1587 raft_consensus.cc:1270] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.083843 2229 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.069610 1728 raft_consensus.cc:2388] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:02.084436 1581 raft_consensus.cc:1270] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.085331 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:02.085841 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:02.082320 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:02.087368 1736 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:02.088949 1736 raft_consensus.cc:1235] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:54:02.055449 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:02.090711 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:02.094847 2229 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:02.094957 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:02.095467 2229 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:02.096613 2567 log.cc:926] Time spent T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20250114 02:54:02.096814 1733 raft_consensus.cc:1270] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.097833 2229 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.099300 1553 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:02.101819 1553 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:02.103626 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:02.104291 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.104512 1578 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
W20250114 02:54:02.104718 2572 log.cc:926] Time spent T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
I20250114 02:54:02.104585 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:02.105484 1580 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:02.105868 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.106225 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:02.106617 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:02.106938 1586 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:02.099334 1660 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:02.107818 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:02.108526 2550 raft_consensus.cc:2798] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:02.108608 1587 raft_consensus.cc:2463] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:02.109138 2550 raft_consensus.cc:695] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:02.082361 1729 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:02.110251 2550 consensus_queue.cc:237] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.112157 2246 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.110248 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:02.122466 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.122934 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.122493 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.124007 1734 raft_consensus.cc:2463] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:54:02.126756 1587 raft_consensus.cc:1270] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.130347 1587 raft_consensus.cc:1235] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:02.131003 1736 raft_consensus.cc:1270] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.098397 1733 raft_consensus.cc:1235] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:02.149134 1727 raft_consensus.cc:2463] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:02.121204 1730 raft_consensus.cc:3054] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.153292 2430 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:02.169052 1736 raft_consensus.cc:1270] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.172410 1736 raft_consensus.cc:1235] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:02.176888 1581 raft_consensus.cc:1270] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.178922 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:02.187036 1733 raft_consensus.cc:1270] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.186910 2242 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:02.191401 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
W20250114 02:54:02.191594 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 89891.3us
I20250114 02:54:02.207744 1735 raft_consensus.cc:2463] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:02.211045 1733 raft_consensus.cc:1270] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.214989 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:02.214776 2242 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.220811 1733 raft_consensus.cc:1270] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 02:54:02.221163 1733 pending_rounds.cc:77] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 1
I20250114 02:54:02.221421 1733 pending_rounds.cc:99] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 02:54:02.221669 1733 raft_consensus.cc:2883] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:02.222146 2242 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.216348 1553 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
W20250114 02:54:02.217100 2573 log.cc:926] Time spent T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
I20250114 02:54:02.223223 1733 raft_consensus.cc:1270] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.217500 2430 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.224713 1733 raft_consensus.cc:1235] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:02.225827 2430 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.226071 1733 raft_consensus.cc:1235] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:02.232110 1553 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:02.233300 2511 raft_consensus.cc:2743] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:02.236330 1587 raft_consensus.cc:1270] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:02.240481 1734 raft_consensus.cc:1235] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:02.245223 1731 raft_consensus.cc:2405] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250114 02:54:02.242812 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.247398 1553 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.240494 2106 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.248625 2430 raft_consensus.cc:2798] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:02.248863 1553 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.250680 2242 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:02.250325 1553 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.251133 2242 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:02.249049 2430 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:02.254822 1586 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:02.259234 2242 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:02.259402 2385 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:02.259804 1482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39748 (request call id 99) took 8626 ms. Trace:
I20250114 02:54:02.260040 2385 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.277033 1727 raft_consensus.cc:1270] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 02:54:02.277422 1727 pending_rounds.cc:77] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 1
I20250114 02:54:02.279430 1727 pending_rounds.cc:99] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 02:54:02.279683 1727 raft_consensus.cc:2883] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:02.262135 2430 raft_consensus.cc:3054] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:02.281639 1729 raft_consensus.cc:1270] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 02:54:02.260207 1482 rpcz_store.cc:276] 0114 02:53:53.633524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:53:53.643834 (+ 10310us) service_pool.cc:225] Handling call
0114 02:53:53.997987 (+354153us) spinlock_profiling.cc:243] Waited 72.7 ms on lock 0x7b54002b02d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605ad50 00007fbce60ddf2a 00007fbcebdf9dec 00007fbcebef1dee 00007fbcea444fc4 00007fbcea444f52 00007fbcea444e81 00007fbcea444dfc 00007fbcea4440b2 00007fbce4500dfc 00007fbce4500236
0114 02:54:02.259592 (+8261605us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:01.101518 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:01.210668 (+109150us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:01.210698 (+ 30us) write_op.cc:435] Acquired schema lock
0114 02:54:01.210713 (+ 15us) tablet.cc:659] Decoding operations
0114 02:54:01.211047 (+ 334us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:01.211100 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:01.211128 (+ 28us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:01.211316 (+ 188us) tablet.cc:698] Row locks acquired
0114 02:54:01.211329 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:01.211457 (+ 128us) write_op.cc:270] Start()
0114 02:54:01.211557 (+ 100us) write_op.cc:276] Timestamp: P: 1736823241211418 usec, L: 0
0114 02:54:01.211571 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:01.212163 (+ 592us) log.cc:844] Serialized 537 byte log entry
0114 02:54:01.467025 (+254862us) op_driver.cc:464] REPLICATION: finished
0114 02:54:01.531886 (+ 64861us) write_op.cc:301] APPLY: starting
0114 02:54:01.531957 (+ 71us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:01.532040 (+ 83us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:01.532173 (+ 133us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:01.532391 (+ 218us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:01.532526 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:01.532571 (+ 45us) write_op.cc:312] APPLY: finished
0114 02:54:01.532747 (+ 176us) log.cc:844] Serialized 22 byte log entry
0114 02:54:01.532902 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:01.532973 (+ 71us) write_op.cc:454] Released schema lock
0114 02:54:01.533006 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:01.533291 (+ 285us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:01.533323 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3337,"reported_tablets":14,"spinlock_wait_cycles":167274752,"child_traces":[["op",{"apply.queue_time_us":64642,"apply.run_cpu_time_us":1391,"apply.run_wall_time_us":1392,"num_ops":1,"prepare.queue_time_us":206600,"prepare.run_cpu_time_us":2579,"prepare.run_wall_time_us":110905,"replication_time_us":255291,"thread_start_us":197624,"threads_started":2}]]}
I20250114 02:54:02.285760 2508 raft_consensus.cc:3054] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:02.253023 1483 catalog_manager.cc:5526] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 2, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 2 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
W20250114 02:54:02.297582 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 47386us
I20250114 02:54:02.300174 1553 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.291723 1729 pending_rounds.cc:77] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 1
I20250114 02:54:02.292564 2385 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.310745 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:02.307114 1729 pending_rounds.cc:99] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 02:54:02.314731 2360 raft_consensus.cc:2798] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:02.315173 2360 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:02.315505 2360 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:02.318850 2302 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:02.319504 2302 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.321614 2302 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.322043 2302 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
W20250114 02:54:02.323124 2569 log.cc:926] Time spent T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.384s user 0.002s sys 0.000s
W20250114 02:54:02.324942 2576 log.cc:926] Time spent T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20250114 02:54:02.337932 2528 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:02.338491 2528 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.339000 1483 catalog_manager.cc:5526] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:02.340448 2528 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:02.340865 1483 catalog_manager.cc:5526] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 3, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 3 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:02.313506 2385 raft_consensus.cc:2798] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:02.313688 1729 raft_consensus.cc:2883] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:02.344467 2385 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:02.345961 1483 catalog_manager.cc:5526] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 1 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:02.322494 2302 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.351317 2385 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
W20250114 02:54:02.361793 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 61752.4us
I20250114 02:54:02.372010 2242 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.378108 1586 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
W20250114 02:54:02.381594 2577 log.cc:926] Time spent T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
W20250114 02:54:02.386091 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 58507.2us
I20250114 02:54:02.387015 1702 leader_election.cc:400] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3.
I20250114 02:54:02.387284 1702 leader_election.cc:403] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250114 02:54:02.388200 1702 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.389402 1702 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.391039 2528 raft_consensus.cc:2798] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:02.391413 2528 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:02.391712 2528 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.392778 2416 raft_consensus.cc:2798] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:02.393180 2416 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:02.393486 2416 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
W20250114 02:54:02.394939 2583 log.cc:926] Time spent T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20250114 02:54:02.397014 2242 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.399047 1586 raft_consensus.cc:1270] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.401048 2360 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.401903 2416 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.401654 2385 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.434098 1585 raft_consensus.cc:1235] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:02.431526 1702 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:02.398519 1730 raft_consensus.cc:2463] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:02.441673 1483 catalog_manager.cc:5526] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 1 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:02.382256 2513 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.435300 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:02.458083 1580 raft_consensus.cc:1235] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:54:02.458120 2512 raft_consensus.cc:2743] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3.
I20250114 02:54:02.459018 2529 raft_consensus.cc:2743] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:02.461961 2385 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.462482 1728 raft_consensus.cc:2388] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:02.453457 2416 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:02.463065 2302 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.461663 2360 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.454180 2246 raft_consensus.cc:2743] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:02.454362 2368 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:02.463702 1728 raft_consensus.cc:1235] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:02.464263 2368 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.466194 2368 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.467721 1728 raft_consensus.cc:1235] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:02.469940 2513 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:02.476621 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 101700us
I20250114 02:54:02.480501 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.481050 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:02.455426 2570 log.cc:926] Time spent T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20250114 02:54:02.483819 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.483536 1483 catalog_manager.cc:5526] T c39189f5c30e4eec95701fe6a370896e P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 3, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 3 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:02.485110 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:02.485388 1483 catalog_manager.cc:5526] T daf552d03bce48dda4e5b9772287649a P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 2, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 2 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:02.487844 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.487787 1483 catalog_manager.cc:5526] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 2, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:02.489041 2528 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.489655 1659 raft_consensus.cc:1270] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.519951 1734 raft_consensus.cc:1235] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:02.490211 1579 raft_consensus.cc:1235] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20250114 02:54:02.523280 2592 log.cc:926] Time spent T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250114 02:54:02.495296 2106 raft_consensus.cc:2743] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:02.521240 1483 catalog_manager.cc:5526] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:02.527046 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:02.501916 2368 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:02.533627 2385 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:02.534314 2385 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.536425 1585 raft_consensus.cc:1235] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:54:02.536631 2385 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.538645 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:02.527194 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:02.547478 1483 catalog_manager.cc:5526] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 2, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 2 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:02.551003 1728 raft_consensus.cc:1270] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250114 02:54:02.551399 1728 pending_rounds.cc:77] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 0
I20250114 02:54:02.551659 1728 pending_rounds.cc:99] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250114 02:54:02.551893 1728 raft_consensus.cc:2883] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:02.555084 2528 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:02.555529 1657 raft_consensus.cc:1270] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.544898 2368 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.563207 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:02.563880 2368 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.565752 1582 raft_consensus.cc:1235] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
W20250114 02:54:02.568985 2596 log.cc:926] Time spent T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.141s user 0.002s sys 0.000s
I20250114 02:54:02.570271 1702 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.559775 2430 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.578012 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:02.548853 2106 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:02.581744 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:02.582769 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:02.584095 1729 raft_consensus.cc:1235] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:02.584123 1587 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:02.585350 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:02.587985 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.581282 2508 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:02.591293 1587 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:02.588871 1578 raft_consensus.cc:1235] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:02.592922 2508 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:02.602746 1587 raft_consensus.cc:1270] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250114 02:54:02.603482 2508 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:02.605743 1732 raft_consensus.cc:1270] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250114 02:54:02.607652 2591 log.cc:926] Time spent T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.226s user 0.004s sys 0.000s
I20250114 02:54:02.611004 1583 raft_consensus.cc:1235] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:02.597088 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:54:02.611336 2588 log.cc:926] Time spent T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.172s user 0.003s sys 0.000s
W20250114 02:54:02.614591 2601 log.cc:926] Time spent T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250114 02:54:02.616303 1586 raft_consensus.cc:1235] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:54:02.594115 2106 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.621423 1583 raft_consensus.cc:1235] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:02.621610 2346 consensus_queue.cc:1035] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:02.622853 1583 raft_consensus.cc:1270] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:02.626542 2346 consensus_queue.cc:1046] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250114 02:54:02.629786 2428 raft_consensus.cc:3049] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:02.629957 1578 raft_consensus.cc:1270] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250114 02:54:02.631289 2587 log.cc:926] Time spent T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.250s user 0.000s sys 0.006s
I20250114 02:54:02.630045 1584 raft_consensus.cc:1235] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:02.629266 1583 raft_consensus.cc:1270] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:02.640770 1702 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.638476 2106 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.641608 2512 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:02.641999 2512 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:02.642376 2512 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.646760 2346 consensus_queue.cc:1035] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:02.648613 2524 consensus_queue.cc:1035] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:02.649446 1702 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:02.650231 2528 raft_consensus.cc:2743] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:02.651590 1702 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:02.630157 2428 raft_consensus.cc:738] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:02.666824 2428 consensus_queue.cc:260] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.668102 2428 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.651506 2524 consensus_queue.cc:1035] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:02.653046 2346 consensus_queue.cc:1035] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:02.653256 2520 consensus_queue.cc:1035] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:02.653422 2197 consensus_queue.cc:1046] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:02.680202 2197 raft_consensus.cc:3049] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:02.680533 2197 raft_consensus.cc:738] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:02.640076 2430 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.627063 1582 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:02.627599 1580 raft_consensus.cc:1235] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:02.662487 1553 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:02.678025 2368 consensus_queue.cc:1046] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:02.683415 2197 consensus_queue.cc:260] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.684743 2528 raft_consensus.cc:2743] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:54:02.685390 2197 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.685799 2430 raft_consensus.cc:2798] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:02.686187 2430 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:02.686537 2430 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:02.688439 2550 consensus_queue.cc:1035] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:02.691565 2385 consensus_queue.cc:1035] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:02.695097 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:02.698966 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 47484.5us
I20250114 02:54:02.699754 1702 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.700708 2528 raft_consensus.cc:2798] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:54:02.701871 2346 raft_consensus.cc:3049] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:54:02.701930 2528 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:02.702333 2346 raft_consensus.cc:738] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:02.702461 2528 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:02.703545 1657 raft_consensus.cc:1270] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:02.703238 2346 consensus_queue.cc:260] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:02.703974 1553 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.106.67:42283 timed out after 1.632s (SENT) [suppressed 54 similar messages]
I20250114 02:54:02.704401 2346 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:02.705932 1656 raft_consensus.cc:1270] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 02:54:02.706445 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.706804 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:02.707113 1661 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:02.707934 2512 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.709906 2512 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:02.711035 2508 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.713548 2508 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:02.721638 1702 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.722313 2528 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.722734 2508 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:02.723289 2508 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:02.723731 2508 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:02.727442 2368 consensus_queue.cc:1046] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:02.728649 2368 consensus_queue.cc:1035] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:02.731842 2609 raft_consensus.cc:3049] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:54:02.731734 2520 consensus_queue.cc:1035] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:02.732227 2609 raft_consensus.cc:738] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:02.733062 2609 consensus_queue.cc:260] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.733947 2520 consensus_queue.cc:1046] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:02.734163 2368 consensus_queue.cc:1046] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:02.734390 2524 consensus_queue.cc:1035] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:02.734841 2609 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:02.734990 2520 raft_consensus.cc:3049] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:02.735069 2368 raft_consensus.cc:3049] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:02.735558 2368 raft_consensus.cc:738] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:02.735637 2520 raft_consensus.cc:738] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:02.736591 2368 consensus_queue.cc:260] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.736757 2520 consensus_queue.cc:260] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.737743 2368 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.738078 2520 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.741264 2430 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.747758 2512 consensus_queue.cc:1035] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:02.749658 2385 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:02.750286 2385 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.755125 1702 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:02.764902 2512 raft_consensus.cc:2743] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:54:02.766816 2528 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625)
I20250114 02:54:02.769656 2385 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.771148 2430 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.771592 1702 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.777858 2508 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.779908 2508 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:02.780721 2528 raft_consensus.cc:2798] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250114 02:54:02.788376 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 84430.5us
W20250114 02:54:02.794651 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 53892.7us
I20250114 02:54:02.799269 1702 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.788383 2528 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:02.807463 2508 raft_consensus.cc:2798] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:02.809648 2508 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:02.810026 2508 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:02.807581 2528 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:02.818323 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:02.840355 1702 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.841890 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:02.842612 1660 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:02.847800 2611 consensus_queue.cc:1046] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:02.848505 1553 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:02.854118 2513 raft_consensus.cc:2798] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:02.850167 2550 consensus_queue.cc:1035] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20250114 02:54:02.859735 2513 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:02.860095 2513 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.846381 2512 raft_consensus.cc:2798] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:02.842478 2428 consensus_queue.cc:1046] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20250114 02:54:02.862335 2512 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:02.850114 2346 consensus_queue.cc:1035] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20250114 02:54:02.889642 2428 consensus_queue.cc:1225] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer b89550620e1d455d887be72232bb5454 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 02:54:02.890311 1660 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:02.891182 2197 raft_consensus.cc:3049] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:02.861970 2106 consensus_queue.cc:1035] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20250114 02:54:02.882248 2512 raft_consensus.cc:3054] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.893718 1553 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:02.891604 2197 raft_consensus.cc:738] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:02.898043 1702 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:02.895663 2508 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.900056 1702 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.901147 1702 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.901342 2508 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:02.897204 2593 raft_consensus.cc:2798] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:02.903836 2368 raft_consensus.cc:3049] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:02.904186 2593 raft_consensus.cc:695] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:02.904242 2368 raft_consensus.cc:738] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:02.905117 2368 consensus_queue.cc:260] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.905597 2528 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.906273 2368 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.906504 2513 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.905112 2593 consensus_queue.cc:237] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.907728 2442 raft_consensus.cc:2743] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:02.908432 2523 raft_consensus.cc:2798] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:54:02.895161 2428 consensus_queue.cc:1035] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20250114 02:54:02.908938 2508 raft_consensus.cc:2798] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:02.910008 1702 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
W20250114 02:54:02.927174 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 44227.4us
I20250114 02:54:02.929021 1483 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39724 (request call id 111) took 1431 ms. Trace:
I20250114 02:54:02.932314 2416 raft_consensus.cc:2743] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:02.932354 1702 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:02.933485 2513 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:02.934990 2416 raft_consensus.cc:2743] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:02.934469 1702 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.925822 2523 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:02.927050 2508 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:02.929402 1483 rpcz_store.cc:276] 0114 02:54:01.495679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:01.495830 (+ 151us) service_pool.cc:225] Handling call
0114 02:54:01.507648 (+ 11818us) spinlock_profiling.cc:243] Waited 4.65 ms on lock 0x7b54002b02d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605ad50 00007fbce60ddf2a 00007fbcebdf9dec 00007fbcebef1dee 00007fbcea444fc4 00007fbcea444f52 00007fbcea444e81 00007fbcea444dfc 00007fbcea4440b2 00007fbce4500dfc 00007fbce4500236
0114 02:54:02.927185 (+1419537us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:02.678389 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:02.689146 (+ 10757us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:02.689195 (+ 49us) write_op.cc:435] Acquired schema lock
0114 02:54:02.689213 (+ 18us) tablet.cc:659] Decoding operations
0114 02:54:02.690219 (+ 1006us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:02.690318 (+ 99us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:02.690348 (+ 30us) tablet.cc:682] Acquiring locks for 10 operations
0114 02:54:02.691221 (+ 873us) tablet.cc:698] Row locks acquired
0114 02:54:02.691242 (+ 21us) write_op.cc:260] PREPARE: finished
0114 02:54:02.691409 (+ 167us) write_op.cc:270] Start()
0114 02:54:02.691529 (+ 120us) write_op.cc:276] Timestamp: P: 1736823242691347 usec, L: 0
0114 02:54:02.691563 (+ 34us) op_driver.cc:348] REPLICATION: starting
0114 02:54:02.692175 (+ 612us) log.cc:844] Serialized 3999 byte log entry
0114 02:54:02.803514 (+111339us) op_driver.cc:464] REPLICATION: finished
0114 02:54:02.831384 (+ 27870us) write_op.cc:301] APPLY: starting
0114 02:54:02.831498 (+ 114us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:02.831841 (+ 343us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:02.832044 (+ 203us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:02.833616 (+ 1572us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:02.833993 (+ 377us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0114 02:54:02.834076 (+ 83us) write_op.cc:312] APPLY: finished
0114 02:54:02.834594 (+ 518us) log.cc:844] Serialized 76 byte log entry
0114 02:54:02.834790 (+ 196us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:02.841909 (+ 7119us) write_op.cc:454] Released schema lock
0114 02:54:02.844570 (+ 2661us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:02.844923 (+ 353us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:02.844954 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":110875,"reported_tablets":31,"spinlock_wait_cycles":10688256,"child_traces":[["op",{"apply.queue_time_us":27584,"apply.run_cpu_time_us":4155,"apply.run_wall_time_us":13557,"num_ops":10,"prepare.queue_time_us":18201,"prepare.run_cpu_time_us":11360,"prepare.run_wall_time_us":71598,"raft.queue_time_us":35518,"raft.run_cpu_time_us":6570,"raft.run_wall_time_us":27777,"replication_time_us":111808,"thread_start_us":116229,"threads_started":4,"wal-append.queue_time_us":71241}]]}
I20250114 02:54:02.946280 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:02.913941 2512 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.957381 1662 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.958639 2528 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:02.941481 2442 raft_consensus.cc:2798] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:02.945641 2523 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:02.953058 2508 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:02.969259 1628 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:02.975176 1702 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:02.983588 1702 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:02.984714 2416 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:02.985780 2528 raft_consensus.cc:2743] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:02.940512 2197 consensus_queue.cc:260] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:02.987483 1702 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:02.991652 2528 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:02.992095 2528 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:02.992519 2528 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:02.995061 1702 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:03.000131 1702 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:02.971390 2520 consensus_queue.cc:1035] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20250114 02:54:03.004909 2512 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:03.019717 2512 raft_consensus.cc:2798] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:03.020164 2512 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:03.020614 2512 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:02.964985 2442 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:03.022994 2442 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:03.005726 2630 consensus_queue.cc:1035] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:03.006353 2615 consensus_queue.cc:1035] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:03.008121 1628 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:03.011026 2550 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:03.016295 2628 log.cc:926] Time spent T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250114 02:54:03.017072 2197 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.032704 1628 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:03.041419 1628 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
W20250114 02:54:03.044412 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 44382.1us
I20250114 02:54:03.065009 1702 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:03.047763 1662 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:03.031287 2346 raft_consensus.cc:2743] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:03.063133 2630 raft_consensus.cc:2798] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:03.044147 2617 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250114 02:54:03.130159 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 58974.1us
I20250114 02:54:03.127769 1553 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:03.032567 2524 consensus_queue.cc:1046] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.094s
I20250114 02:54:03.031610 2428 consensus_queue.cc:1035] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.097s
I20250114 02:54:03.030543 2523 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.034053 2615 consensus_queue.cc:1046] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.104s
I20250114 02:54:03.129164 2630 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:03.144820 2633 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:03.075205 2625 raft_consensus.cc:2743] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:03.032972 2550 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.071029 2368 consensus_queue.cc:1035] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.078s
I20250114 02:54:03.131978 2617 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:03.138923 2246 raft_consensus.cc:2798] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:03.031910 2629 consensus_queue.cc:1046] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.111s
W20250114 02:54:03.164840 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 99911.1us
I20250114 02:54:03.166388 1702 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:03.152936 2630 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.162528 2617 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.159814 2346 raft_consensus.cc:3049] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:03.176699 2644 consensus_queue.cc:1046] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:03.178951 1702 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:03.151918 2416 raft_consensus.cc:695] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:03.155177 2634 consensus_queue.cc:1046] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20250114 02:54:03.183095 2416 consensus_queue.cc:237] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.184309 2442 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.185493 2550 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:03.168262 2246 raft_consensus.cc:695] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:03.149364 2508 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.171694 2629 consensus_queue.cc:1225] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer 62daa58e8ff64698bb46ea3fd5290cdd log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 02:54:03.166280 2524 consensus_queue.cc:1035] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20250114 02:54:03.204497 2611 raft_consensus.cc:3049] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:03.205052 2611 raft_consensus.cc:738] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:03.207980 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:03.208798 1660 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:03.186230 2442 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:03.180984 2346 raft_consensus.cc:738] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:03.191052 2640 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:03.196111 2645 raft_consensus.cc:2798] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:03.166131 2633 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:03.226306 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 47499.5us
I20250114 02:54:03.230613 1702 leader_election.cc:304] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:03.225433 2442 raft_consensus.cc:2798] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:03.236815 2246 consensus_queue.cc:237] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.237249 2609 raft_consensus.cc:3049] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:03.206094 2611 consensus_queue.cc:260] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:03.237275 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:54:03.237812 2520 consensus_queue.cc:1035] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:03.225981 2640 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:03.224174 2641 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:03.236261 2625 consensus_queue.cc:1035] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20250114 02:54:03.228624 2645 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:03.239176 2368 raft_consensus.cc:3049] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:03.246938 2368 raft_consensus.cc:738] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:03.247195 2649 raft_consensus.cc:2743] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:03.247767 2368 consensus_queue.cc:260] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.242056 2609 raft_consensus.cc:738] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:03.245412 2640 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.249598 2609 consensus_queue.cc:260] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.250767 2609 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.245898 2641 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.252357 2645 raft_consensus.cc:3054] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.246542 2523 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:03.252498 2633 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:03.250690 2442 raft_consensus.cc:695] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:03.245997 2508 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:03.244982 1702 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:03.259969 2641 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:03.243713 2642 raft_consensus.cc:2743] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:03.276899 1553 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:03.280292 1702 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:03.286665 2641 raft_consensus.cc:2743] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:03.279729 2508 raft_consensus.cc:2743] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250114 02:54:03.291078 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 66122.4us
I20250114 02:54:03.268344 2611 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.300333 1702 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:03.244791 2528 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.239959 2346 consensus_queue.cc:260] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.306500 2197 raft_consensus.cc:3049] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:03.304721 1628 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:03.291543 2513 raft_consensus.cc:2798] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:03.291939 2647 raft_consensus.cc:491] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:03.314422 2416 raft_consensus.cc:2743] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:03.315119 2442 consensus_queue.cc:237] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.316900 2368 raft_consensus.cc:3054] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.348899 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:03.353750 1736 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
W20250114 02:54:03.354183 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 53962us
I20250114 02:54:03.354195 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:03.355422 1587 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:03.313741 1483 catalog_manager.cc:5526] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 3, leader changed from <none> to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 3 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:03.327874 2512 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.330973 2346 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:03.322985 2634 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:03.319397 2513 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:03.373976 2513 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.374353 1732 raft_consensus.cc:1235] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:03.374423 1586 raft_consensus.cc:1270] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:03.334666 2647 raft_consensus.cc:513] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.361472 1734 raft_consensus.cc:1270] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:03.310783 2197 raft_consensus.cc:738] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:03.367924 2640 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.368417 1727 raft_consensus.cc:1270] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:03.368964 1733 raft_consensus.cc:1270] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:03.369210 1729 raft_consensus.cc:1235] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:03.369956 1580 raft_consensus.cc:1235] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:03.372643 1728 raft_consensus.cc:1270] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:03.373524 1730 raft_consensus.cc:1235] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:03.409600 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:03.354033 2528 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:03.410485 1732 raft_consensus.cc:1235] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:03.466511 1730 raft_consensus.cc:1235] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:03.386946 1702 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:03.398109 1483 catalog_manager.cc:5526] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 2, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:03.473635 1587 raft_consensus.cc:2463] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
W20250114 02:54:03.479327 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 92490.4us
I20250114 02:54:03.378245 2634 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:03.498265 1483 catalog_manager.cc:5526] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 0 to 2, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:03.500747 1483 catalog_manager.cc:5526] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:03.503520 1735 raft_consensus.cc:1235] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:03.502477 1483 catalog_manager.cc:5526] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:03.524405 1727 raft_consensus.cc:1235] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:03.499678 2634 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.485110 1581 raft_consensus.cc:1270] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:03.485419 2528 raft_consensus.cc:2743] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:03.409708 2630 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.547870 1727 raft_consensus.cc:1235] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:03.533094 1730 raft_consensus.cc:1235] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:03.547847 2416 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:03.550832 1730 raft_consensus.cc:1235] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:03.551115 2416 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.552073 1730 raft_consensus.cc:1235] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:03.552374 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:03.533042 2513 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.553139 2416 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:03.554912 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 63432us
I20250114 02:54:03.555167 1628 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:03.521672 1483 catalog_manager.cc:5526] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 2 to 4, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 4 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:03.554477 2513 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:03.562760 1483 catalog_manager.cc:5526] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 2, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:03.565066 1483 catalog_manager.cc:5526] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:03.566612 1483 catalog_manager.cc:5526] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 3, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:03.568336 1483 catalog_manager.cc:5526] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 3, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:03.570309 1483 catalog_manager.cc:5526] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 3, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 3 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:03.558367 2627 raft_consensus.cc:2798] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:03.572129 1702 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:03.572026 1483 catalog_manager.cc:5526] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 3, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 3 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:03.575863 1702 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:03.574113 1483 catalog_manager.cc:5526] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:03.578469 1483 catalog_manager.cc:5526] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 2 to 4, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 4 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:03.573678 2627 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:03.583575 1483 catalog_manager.cc:5526] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 3, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:03.562402 2625 consensus_queue.cc:1046] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20250114 02:54:03.577207 2524 consensus_queue.cc:1046] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20250114 02:54:03.568699 2643 consensus_queue.cc:1046] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20250114 02:54:03.574177 2651 consensus_queue.cc:1035] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20250114 02:54:03.591405 2428 consensus_queue.cc:1046] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.005s
I20250114 02:54:03.582644 2512 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:03.596875 2630 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:03.598682 2627 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:03.576193 2644 consensus_queue.cc:1035] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20250114 02:54:03.578020 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.582530 1728 raft_consensus.cc:1235] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20250114 02:54:03.573216 2629 consensus_queue.cc:1035] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250114 02:54:03.568255 2520 consensus_queue.cc:1046] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250114 02:54:03.563835 2646 consensus_queue.cc:1035] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20250114 02:54:03.592931 2633 consensus_queue.cc:1046] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20250114 02:54:03.597594 2640 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:03.612226 2528 raft_consensus.cc:2743] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:03.582384 2416 raft_consensus.cc:2743] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:54:03.622104 2508 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:03.591782 2645 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.611739 2647 leader_election.cc:290] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:03.634634 1656 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.633989 2346 raft_consensus.cc:3049] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:03.601871 2617 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.659718 2650 consensus_queue.cc:1046] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20250114 02:54:03.661195 2655 consensus_queue.cc:1046] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.086s
I20250114 02:54:03.662810 1727 raft_consensus.cc:1235] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:03.661489 2617 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:03.635699 2615 consensus_queue.cc:1046] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250114 02:54:03.649912 2630 raft_consensus.cc:3049] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:03.717707 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.633430 2513 consensus_queue.cc:1035] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
W20250114 02:54:03.703996 2658 log.cc:926] Time spent T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20250114 02:54:03.656316 2346 raft_consensus.cc:738] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:03.703780 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:03.704051 2647 raft_consensus.cc:3049] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:03.689236 2428 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:03.733357 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:03.646703 2197 consensus_queue.cc:260] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.720643 2630 raft_consensus.cc:738] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:03.688994 2645 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:03.664299 2508 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.731034 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:03.737247 2617 raft_consensus.cc:3049] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
W20250114 02:54:03.745709 2654 log.cc:926] Time spent T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
I20250114 02:54:03.735230 2647 raft_consensus.cc:738] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:03.762900 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
W20250114 02:54:03.763331 2656 log.cc:926] Time spent T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
W20250114 02:54:03.764703 2660 log.cc:926] Time spent T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20250114 02:54:03.765595 1728 raft_consensus.cc:2463] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:54:03.768019 2629 raft_consensus.cc:3049] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:03.771229 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.772624 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.803370 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.804518 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.773905 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
W20250114 02:54:03.775064 2659 log.cc:926] Time spent T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.151s user 0.000s sys 0.000s
I20250114 02:54:03.806540 2625 raft_consensus.cc:3049] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:03.807026 2625 raft_consensus.cc:738] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:03.775426 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.807736 2634 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.807997 2625 consensus_queue.cc:260] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.809329 2625 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.809715 2634 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:03.810458 2663 log.cc:926] Time spent T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20250114 02:54:03.787204 1662 raft_consensus.cc:1270] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:03.749392 2430 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:03.774403 2627 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.793701 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:03.794469 1582 raft_consensus.cc:1270] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 02:54:03.795184 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:03.794898 2630 consensus_queue.cc:260] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:03.796712 2667 log.cc:926] Time spent T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.056s user 0.003s sys 0.000s
I20250114 02:54:03.776784 2617 raft_consensus.cc:738] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:03.791949 2629 raft_consensus.cc:738] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:03.798178 1733 raft_consensus.cc:1235] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:03.783921 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.760879 2197 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:03.792243 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.813879 2646 raft_consensus.cc:3049] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:03.813913 1654 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:03.820112 2645 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:03.821177 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.821676 1729 raft_consensus.cc:1270] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 02:54:03.821815 1581 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.819748 1582 pending_rounds.cc:77] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 1
I20250114 02:54:03.824528 1585 raft_consensus.cc:2463] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:03.828928 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.831337 1660 raft_consensus.cc:2405] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:03.830530 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:54:03.840034 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 195960us
I20250114 02:54:03.839946 1727 raft_consensus.cc:2463] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:03.738538 2428 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.822901 2416 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:03.831878 2633 consensus_queue.cc:1225] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer b89550620e1d455d887be72232bb5454 log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250114 02:54:03.875809 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:03.874552 2416 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.823231 2430 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.856551 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:03.852787 1582 pending_rounds.cc:99] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 02:54:03.887100 1582 raft_consensus.cc:2883] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:03.854794 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.892637 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:03.894009 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:03.897712 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:03.837852 2646 raft_consensus.cc:738] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:03.852875 2630 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.853277 2645 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.886636 2617 consensus_queue.cc:260] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.790970 2346 consensus_queue.cc:260] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.917273 1628 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:03.916184 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.919234 2650 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:03.922544 1727 raft_consensus.cc:2388] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
W20250114 02:54:03.925690 2662 log.cc:926] Time spent T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.107s user 0.000s sys 0.001s
I20250114 02:54:03.915733 2651 raft_consensus.cc:3049] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
W20250114 02:54:03.926990 2611 consensus_meta.cc:287] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.622s user 0.017s sys 0.001s
W20250114 02:54:03.927641 2368 consensus_meta.cc:287] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.580s user 0.011s sys 0.001s
I20250114 02:54:03.915763 2674 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:03.946300 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:03.838116 2647 consensus_queue.cc:260] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.927057 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:03.925386 1730 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:03.942665 1733 raft_consensus.cc:2388] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:03.948446 2611 raft_consensus.cc:2798] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:03.952523 2368 consensus_queue.cc:1046] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250114 02:54:03.936118 2246 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:03.955871 2627 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:03.932220 1656 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:03.942680 2550 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:03.957844 1581 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:03.959273 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:03.959457 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:03.938203 1578 raft_consensus.cc:1235] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:03.963174 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:03.965749 2611 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:03.944710 2650 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.942653 2651 raft_consensus.cc:738] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:03.927394 2653 consensus_queue.cc:1035] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.022s
I20250114 02:54:03.973879 1661 raft_consensus.cc:1235] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:03.957865 1659 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:03.975960 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:03.980568 1727 raft_consensus.cc:2388] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:03.899089 2629 consensus_queue.cc:260] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.985018 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:03.974220 2246 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.986478 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:03.986811 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:03.987609 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.036674 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:04.037248 1583 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.037600 1729 raft_consensus.cc:2388] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:04.039016 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:03.989349 2611 raft_consensus.cc:3054] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:03.975715 2428 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.043192 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:03.978749 2550 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.044051 1653 raft_consensus.cc:2363] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3.
W20250114 02:54:04.044464 2661 log.cc:926] Time spent T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.309s user 0.001s sys 0.001s
I20250114 02:54:04.045271 2550 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.046399 2246 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.046979 1587 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:04.047189 1734 raft_consensus.cc:1270] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:04.048307 1734 raft_consensus.cc:1270] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:04.049103 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.049438 2430 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.049803 1587 raft_consensus.cc:2388] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:04.050405 2416 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:04.030949 1735 raft_consensus.cc:2388] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:04.031991 1731 raft_consensus.cc:2388] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:04.036237 2629 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.036793 1582 raft_consensus.cc:2388] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:03.972141 2615 consensus_queue.cc:1035] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20250114 02:54:04.052953 1734 raft_consensus.cc:1270] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250114 02:54:04.038396 2669 log.cc:926] Time spent T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20250114 02:54:03.953372 2646 consensus_queue.cc:260] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.050485 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.063730 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:04.064340 1734 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.065238 2246 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:04.065829 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.066749 2651 consensus_queue.cc:260] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:03.949790 2674 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.062808 2430 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:04.056771 2508 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:04.056715 2609 raft_consensus.cc:3049] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:04.029572 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:04.065898 2246 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.122295 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.094892 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.123102 1587 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.096877 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.098618 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.100093 2651 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.100592 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:04.100076 1734 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:04.103282 1582 raft_consensus.cc:1235] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:54:04.106247 2645 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:04.092921 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:04.111291 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.061262 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.128552 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.128978 1727 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:04.124208 1732 raft_consensus.cc:2388] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 2: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:04.125769 1735 raft_consensus.cc:2371] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3. Re-sending same reply.
I20250114 02:54:04.129155 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.130658 1585 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.110710 2653 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.132730 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:04.112797 2609 raft_consensus.cc:738] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:04.137876 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.137951 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.134732 1732 raft_consensus.cc:2371] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3. Re-sending same reply.
I20250114 02:54:04.138756 1734 raft_consensus.cc:2388] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:04.139765 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:04.139684 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.140519 1727 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.134687 2385 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.143586 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:04.144817 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.143062 2674 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.121637 2430 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.153131 1734 raft_consensus.cc:1270] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:04.155742 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.158516 1732 raft_consensus.cc:2388] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:04.142918 2385 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.162169 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:04.163671 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:04.166752 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 228072us
I20250114 02:54:04.170388 1662 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
W20250114 02:54:04.176367 2681 log.cc:926] Time spent T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20250114 02:54:04.183413 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.184929 2647 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.185050 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.185882 1579 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
W20250114 02:54:04.187029 2683 log.cc:926] Time spent T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250114 02:54:04.187258 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.188452 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.188999 1582 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.189503 2650 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.190490 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:04.191582 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:04.196105 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.196329 2246 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.197527 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.141036 2653 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.201704 1653 raft_consensus.cc:2405] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:04.213795 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.214850 1734 raft_consensus.cc:2388] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:04.190801 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.170823 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
W20250114 02:54:04.225606 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 84821.6us
I20250114 02:54:04.188997 2609 consensus_queue.cc:260] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.234123 1654 raft_consensus.cc:1270] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:04.235280 2692 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.226779 2385 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.236261 1553 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
W20250114 02:54:04.238209 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 64742.4us
I20250114 02:54:04.245352 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:04.247526 2685 raft_consensus.cc:2798] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:04.248059 2685 raft_consensus.cc:695] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:04.248886 2685 consensus_queue.cc:237] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.249970 1585 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:04.242182 2692 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.251197 1653 raft_consensus.cc:1235] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:04.252113 2611 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.252672 1661 raft_consensus.cc:1235] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:04.252807 2692 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.253960 2615 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.254436 1659 raft_consensus.cc:1235] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:04.254413 2615 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.256525 2615 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.257920 1654 raft_consensus.cc:2463] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:04.261134 1587 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:04.262986 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.263470 1727 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:04.240048 2653 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.264020 1587 raft_consensus.cc:2388] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:04.264343 1734 raft_consensus.cc:1270] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:04.265846 1583 raft_consensus.cc:2405] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:04.266477 1581 raft_consensus.cc:1235] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:04.266683 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.266932 2611 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.266978 2106 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.268119 1585 raft_consensus.cc:2388] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:04.268059 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.269028 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:04.269769 1581 raft_consensus.cc:1235] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:04.272100 2617 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.274014 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:04.274832 1659 raft_consensus.cc:2388] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:04.276531 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:04.277320 2684 log.cc:926] Time spent T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.162s user 0.002s sys 0.000s
I20250114 02:54:04.278358 1659 raft_consensus.cc:1270] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:04.278880 1655 raft_consensus.cc:2463] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:04.279747 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:04.280046 2346 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.235532 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:04.281646 1584 raft_consensus.cc:2463] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:04.282647 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:04.283794 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:04.285315 1584 raft_consensus.cc:1270] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:04.287377 2646 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.288970 1585 raft_consensus.cc:1235] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:04.292533 1580 raft_consensus.cc:1270] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:04.293692 1628 leader_election.cc:304] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:04.295065 2633 raft_consensus.cc:2798] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:04.295509 2633 raft_consensus.cc:491] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:04.295903 2633 raft_consensus.cc:3054] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.228209 2430 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.267969 2106 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.301497 2106 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.301972 2106 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.265511 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:04.307986 1702 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:04.314669 1702 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:04.323834 1702 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:04.240583 2609 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.326812 1702 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:04.324633 2645 raft_consensus.cc:2798] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:04.329047 2508 raft_consensus.cc:2798] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:04.328020 2642 raft_consensus.cc:2798] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:04.329190 2645 raft_consensus.cc:695] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:04.329548 2508 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:04.329806 2642 raft_consensus.cc:695] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:04.329995 2508 raft_consensus.cc:3054] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.330263 2645 consensus_queue.cc:237] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.330662 2642 consensus_queue.cc:237] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.334743 2528 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.335173 2528 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.335825 1658 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:04.337047 2528 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:04.337239 2508 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:04.339354 2625 consensus_meta.cc:287] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.530s user 0.001s sys 0.011s
I20250114 02:54:04.341094 1656 raft_consensus.cc:2405] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:04.341470 1578 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:04.342130 2593 consensus_queue.cc:1035] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:04.346181 1553 leader_election.cc:304] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:04.346921 2593 raft_consensus.cc:2798] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:04.347261 2593 raft_consensus.cc:491] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:04.347584 2593 raft_consensus.cc:3054] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.348769 1657 raft_consensus.cc:2405] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:04.352185 2430 consensus_queue.cc:1035] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:04.352300 1553 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
W20250114 02:54:04.356755 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 61954.9us
I20250114 02:54:04.353225 2508 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:04.364653 1702 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:04.302392 2106 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.365218 1628 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:04.367403 2106 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.369432 2625 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.370116 2625 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.392613 2611 raft_consensus.cc:2798] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
W20250114 02:54:04.407362 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 55160.5us
I20250114 02:54:04.395047 2106 raft_consensus.cc:2798] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:04.398120 2508 raft_consensus.cc:2798] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:54:04.383901 2647 consensus_queue.cc:1035] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20250114 02:54:04.392594 2640 consensus_queue.cc:1035] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20250114 02:54:04.412797 2611 raft_consensus.cc:695] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:04.392978 2416 consensus_queue.cc:1046] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20250114 02:54:04.394052 2512 consensus_queue.cc:1035] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.016s
I20250114 02:54:04.415800 1553 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.418965 2629 consensus_queue.cc:1046] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:04.419721 1583 raft_consensus.cc:1235] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:04.416934 1579 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:04.420886 1585 raft_consensus.cc:1235] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:04.409811 2106 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:04.426061 2625 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.426921 2106 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.428489 2512 raft_consensus.cc:3049] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:04.428975 2512 raft_consensus.cc:738] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:04.431905 2593 raft_consensus.cc:513] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.433813 2593 leader_election.cc:290] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.438973 1660 raft_consensus.cc:2405] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:04.439059 1553 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:04.440260 1553 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.442128 1553 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.415182 2508 raft_consensus.cc:695] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:04.446245 2593 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.450742 2633 raft_consensus.cc:513] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.451344 2508 consensus_queue.cc:237] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.453248 2633 leader_election.cc:290] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.443153 2442 raft_consensus.cc:2798] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:04.444921 2246 raft_consensus.cc:2743] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:04.445713 1553 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:04.445607 2513 consensus_queue.cc:1035] T 3a7e023d568848b9861c2976a138d461 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:04.446398 2611 consensus_queue.cc:237] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.449113 1582 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:04.437213 2430 raft_consensus.cc:2743] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250114 02:54:04.454860 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 90271.9us
I20250114 02:54:04.429883 2512 consensus_queue.cc:260] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.458479 1553 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.459676 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.463764 1728 raft_consensus.cc:1235] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:04.465090 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:04.469648 1735 raft_consensus.cc:1235] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20250114 02:54:04.471302 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.502449 1727 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:04.481129 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:04.503387 1587 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:04.482430 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.484166 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:04.484997 1732 raft_consensus.cc:1235] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:04.508256 1580 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:04.485569 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.508955 1736 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
W20250114 02:54:04.509007 2699 log.cc:926] Time spent T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20250114 02:54:04.493126 1702 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.494949 1553 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.455804 2645 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:04.495596 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.495862 2512 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.496526 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:04.510876 1553 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.516995 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:04.517315 1553 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.450749 2692 raft_consensus.cc:2798] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:04.493073 2385 raft_consensus.cc:2798] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250114 02:54:04.515349 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:54:04.524317 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.528702 1735 raft_consensus.cc:2463] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:04.525688 1628 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:04.500278 2246 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:04.495489 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:04.536226 1702 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.543752 1702 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.545082 1702 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.504881 1579 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.559203 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:04.558882 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.560626 1727 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:04.560741 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:04.561306 1735 raft_consensus.cc:3054] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.513912 1584 raft_consensus.cc:1235] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:04.564858 1732 raft_consensus.cc:1235] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:04.569303 1731 raft_consensus.cc:1235] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:04.514516 2685 raft_consensus.cc:2743] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:04.528120 2692 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:04.589799 2692 raft_consensus.cc:3054] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.562222 2642 raft_consensus.cc:2743] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:54:04.591600 2642 raft_consensus.cc:2743] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:04.539341 2528 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.538779 2550 raft_consensus.cc:2743] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:04.592783 2528 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.571930 2430 consensus_queue.cc:1035] T 0ecc21b2d4ec4b0aab2d25073324044b P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250114 02:54:04.595367 2528 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:04.596020 2528 raft_consensus.cc:2743] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:04.518540 2645 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.597937 2593 raft_consensus.cc:2743] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:04.599310 2644 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.599606 1628 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:04.599934 2690 raft_consensus.cc:2743] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:04.599789 2644 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.600661 1628 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:04.601583 2644 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.601645 1628 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:04.602602 2644 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:04.602730 2617 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:04.603118 2617 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:04.603448 2617 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.604436 2644 raft_consensus.cc:695] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:04.605562 2644 consensus_queue.cc:237] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.536257 2385 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:04.526957 1583 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:04.560038 2640 raft_consensus.cc:2743] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:04.560382 2680 consensus_queue.cc:1035] T c1f2bc8299cf4834bebd9aa5ae95a56e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20250114 02:54:04.613866 1628 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:04.587879 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.558812 2647 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.539021 2416 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:04.608553 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 63578us
I20250114 02:54:04.621368 1702 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.606779 2650 raft_consensus.cc:2798] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:04.576443 2646 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:04.562582 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:04.568499 1581 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:04.636986 1582 raft_consensus.cc:2388] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:04.638952 2646 raft_consensus.cc:695] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:04.640189 2646 consensus_queue.cc:237] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.641368 2442 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:04.641878 2442 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.613420 2655 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.643353 2655 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.624576 1628 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:04.645800 2655 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.646118 1628 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:04.646405 2645 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:04.623263 1702 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.611169 2385 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.624579 1585 raft_consensus.cc:3054] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.629034 1579 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:04.623581 2647 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.623731 2416 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.624028 2633 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:04.630857 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.606837 2106 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:04.666795 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 44282.1us
I20250114 02:54:04.686165 1702 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.624727 2650 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:04.692070 1702 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.655975 2720 raft_consensus.cc:2743] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:04.665416 2630 raft_consensus.cc:2798] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:04.657388 2655 raft_consensus.cc:2798] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:04.726336 1587 raft_consensus.cc:2388] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:04.678869 2645 raft_consensus.cc:2743] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:04.678603 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:04.679221 2633 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:04.729171 1702 leader_election.cc:400] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3.
I20250114 02:54:04.690306 2650 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.684803 2197 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.732626 1585 raft_consensus.cc:2463] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:04.664811 1735 raft_consensus.cc:2463] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:04.729854 2528 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election won for term 3
W20250114 02:54:04.736816 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 208906us
I20250114 02:54:04.738113 1702 leader_election.cc:304] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.728834 2512 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.718837 2710 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:04.730425 2630 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:04.700628 2640 consensus_queue.cc:1035] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20250114 02:54:04.732697 2655 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:04.740103 2633 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.726231 2611 consensus_queue.cc:1046] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.018s
I20250114 02:54:04.684121 2609 consensus_queue.cc:1046] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20250114 02:54:04.748091 2630 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.742990 1583 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:04.747829 2512 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.752279 2655 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.807057 2442 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.808291 2416 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:04.788601 2508 consensus_queue.cc:1046] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.021s
I20250114 02:54:04.744570 2197 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.764950 2720 raft_consensus.cc:2743] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:04.761752 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:04.748275 2645 consensus_queue.cc:1046] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
W20250114 02:54:04.818398 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 76637.7us
I20250114 02:54:04.824579 2593 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.825174 2512 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:04.825625 2629 raft_consensus.cc:3049] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:04.832748 2246 raft_consensus.cc:2743] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:04.810369 2385 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.840368 2629 raft_consensus.cc:738] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:04.860126 1553 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.863119 1553 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.825826 2106 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.865694 1586 raft_consensus.cc:1235] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:04.862407 2640 raft_consensus.cc:3049] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:04.869448 1553 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.876376 1553 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
W20250114 02:54:04.863584 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 180257us
I20250114 02:54:04.863308 2647 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:04.877463 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 132366us
I20250114 02:54:04.879277 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.881402 1702 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.880165 1734 raft_consensus.cc:2463] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:04.864830 2512 raft_consensus.cc:3049] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:04.858760 2593 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.883698 1702 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.864785 2685 raft_consensus.cc:2743] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:04.872295 2246 raft_consensus.cc:2743] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:04.876049 2640 raft_consensus.cc:738] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:04.884464 2645 consensus_queue.cc:1046] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20250114 02:54:04.875762 2106 raft_consensus.cc:2743] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:04.892252 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:04.912420 1734 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:04.888836 1702 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.864624 2528 raft_consensus.cc:695] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:04.890235 2416 raft_consensus.cc:2743] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250114 02:54:04.898154 2717 log.cc:926] Time spent T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.111s user 0.000s sys 0.001s
I20250114 02:54:04.882409 2629 consensus_queue.cc:260] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:04.893076 2725 log.cc:926] Time spent T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20250114 02:54:04.905603 2385 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.876834 2643 raft_consensus.cc:3049] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:04.909852 1553 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.910671 1628 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:04.920782 1702 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.873958 2690 consensus_queue.cc:1035] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.019s
I20250114 02:54:04.885442 2512 raft_consensus.cc:738] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:04.922977 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:04.925449 2733 raft_consensus.cc:2798] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:04.925565 1579 raft_consensus.cc:2463] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:04.925958 2733 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:04.926367 2733 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.926716 2629 raft_consensus.cc:3054] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.921936 2720 raft_consensus.cc:2743] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:04.926740 1729 raft_consensus.cc:1235] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:04.929102 2692 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.912753 2685 raft_consensus.cc:2743] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:04.931053 2692 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.931942 2645 consensus_queue.cc:1035] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:04.932531 2645 consensus_queue.cc:1225] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer 1e5870f8269b49cf81dc39f1a5c8f5c1 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 02:54:04.933499 2645 raft_consensus.cc:3049] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:04.933831 2645 raft_consensus.cc:738] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:04.923362 1584 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:04.885871 2721 raft_consensus.cc:2798] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:04.919987 2513 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
W20250114 02:54:04.959662 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 49877.7us
I20250114 02:54:04.926291 2617 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.965256 1553 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.965363 2693 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:04.966459 2617 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.967825 2593 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.970862 2651 raft_consensus.cc:3049] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:54:04.934531 2645 consensus_queue.cc:260] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.971236 2651 raft_consensus.cc:738] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:04.971484 2730 raft_consensus.cc:2743] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:04.971825 2645 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.959121 1628 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:04.972023 2651 consensus_queue.cc:260] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.973215 2651 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:04.973140 1628 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:04.974745 2647 raft_consensus.cc:2798] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:04.975296 2647 raft_consensus.cc:695] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:04.975586 2650 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.976384 2647 consensus_queue.cc:237] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.978132 2650 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.978852 2728 raft_consensus.cc:2798] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:04.979413 2728 raft_consensus.cc:695] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:04.979516 2674 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:04.979974 2674 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.980208 2728 consensus_queue.cc:237] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.981782 2674 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.959797 2442 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:04.924577 2643 raft_consensus.cc:738] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:04.983296 1553 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:04.983592 2643 consensus_queue.cc:260] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.984439 1553 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.984663 2643 raft_consensus.cc:3054] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:04.959749 2733 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.987603 2528 consensus_queue.cc:237] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.988812 2733 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.989598 1736 raft_consensus.cc:1235] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20250114 02:54:04.991146 2655 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.943037 2512 consensus_queue.cc:260] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.993050 2655 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.993206 2512 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:04.993849 1628 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:04.994885 2690 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election won for term 3
W20250114 02:54:04.993891 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 73229.4us
I20250114 02:54:04.998899 1702 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.999681 1702 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:05.000680 1702 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:05.001667 2730 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.003733 2633 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.005647 2633 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.005529 2630 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:04.995527 2690 raft_consensus.cc:695] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:05.008348 1702 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:05.009789 1553 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:04.937855 2721 raft_consensus.cc:695] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:04.940230 2513 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.014192 1628 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:04.938982 2416 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:04.939793 2197 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:04.935268 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.037824 1483 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39724 (request call id 112) took 1805 ms. Trace:
I20250114 02:54:05.025038 1702 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:05.027218 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.028406 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.028540 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.040011 1735 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.039815 1628 leader_election.cc:304] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:04.995960 2642 raft_consensus.cc:2743] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:05.050026 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.049183 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.011379 2611 raft_consensus.cc:2798] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:05.009984 2508 raft_consensus.cc:2743] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:05.012012 2720 raft_consensus.cc:2743] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:05.009892 2732 raft_consensus.cc:2798] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:05.046969 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:05.059693 1587 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.011216 2734 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:05.035054 2593 raft_consensus.cc:2743] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:05.035871 2733 raft_consensus.cc:2798] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:05.065021 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.065696 1736 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.063665 1660 raft_consensus.cc:1235] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
W20250114 02:54:05.067036 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 57334.7us
I20250114 02:54:05.067719 1553 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:05.068863 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.069708 1731 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:05.048013 2442 raft_consensus.cc:2743] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:05.038203 1483 rpcz_store.cc:276] 0114 02:54:03.232054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:03.232228 (+ 174us) service_pool.cc:225] Handling call
0114 02:54:05.037767 (+1805539us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:03.834393 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:03.879210 (+ 44817us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:03.879256 (+ 46us) write_op.cc:435] Acquired schema lock
0114 02:54:03.879277 (+ 21us) tablet.cc:659] Decoding operations
0114 02:54:03.948452 (+ 69175us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:03.948511 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:03.952354 (+ 3843us) tablet.cc:682] Acquiring locks for 15 operations
0114 02:54:03.955420 (+ 3066us) tablet.cc:698] Row locks acquired
0114 02:54:03.955443 (+ 23us) write_op.cc:260] PREPARE: finished
0114 02:54:03.955576 (+ 133us) write_op.cc:270] Start()
0114 02:54:03.955694 (+ 118us) write_op.cc:276] Timestamp: P: 1736823243955536 usec, L: 0
0114 02:54:03.955714 (+ 20us) op_driver.cc:348] REPLICATION: starting
0114 02:54:03.956324 (+ 610us) log.cc:844] Serialized 5928 byte log entry
0114 02:54:04.241518 (+285194us) op_driver.cc:464] REPLICATION: finished
0114 02:54:04.512097 (+270579us) write_op.cc:301] APPLY: starting
0114 02:54:04.518372 (+ 6275us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:04.607022 (+ 88650us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:04.619128 (+ 12106us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:04.632204 (+ 13076us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:04.632504 (+ 300us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0114 02:54:04.632561 (+ 57us) write_op.cc:312] APPLY: finished
0114 02:54:04.632965 (+ 404us) log.cc:844] Serialized 106 byte log entry
0114 02:54:04.633787 (+ 822us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:04.633955 (+ 168us) write_op.cc:454] Released schema lock
0114 02:54:04.634010 (+ 55us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:04.636163 (+ 2153us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:04.636198 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":251551,"apply.run_cpu_time_us":4424,"apply.run_wall_time_us":134911,"num_ops":15,"prepare.queue_time_us":183813,"prepare.run_cpu_time_us":27809,"prepare.run_wall_time_us":320453,"raft.queue_time_us":63469,"raft.run_cpu_time_us":11600,"raft.run_wall_time_us":168208,"replication_time_us":285663,"thread_start_us":502695,"threads_started":4,"wal-append.queue_time_us":219131}]]}
I20250114 02:54:05.071805 1727 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:05.017181 2650 consensus_queue.cc:1046] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
I20250114 02:54:04.939328 2640 consensus_queue.cc:260] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.074524 1735 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:05.082152 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:05.064224 2710 raft_consensus.cc:695] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:05.054383 2611 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:05.055665 2625 raft_consensus.cc:2798] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:05.056268 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.054196 1578 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:05.059186 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.065316 2733 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:05.042872 2416 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.089679 1553 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:05.107615 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:05.078828 2724 raft_consensus.cc:2743] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:05.086552 2611 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.075842 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:05.086990 2642 consensus_queue.cc:1046] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20250114 02:54:05.012763 2721 consensus_queue.cc:237] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:05.162809 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 77747us
I20250114 02:54:05.158953 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:05.103178 2732 raft_consensus.cc:695] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:05.208897 1731 raft_consensus.cc:2463] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:05.215018 1736 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:05.200734 1587 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:05.086665 2690 consensus_queue.cc:237] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.086550 2734 raft_consensus.cc:695] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
W20250114 02:54:05.303804 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 214196us
I20250114 02:54:05.304289 1628 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
W20250114 02:54:05.232887 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 170566us
I20250114 02:54:05.104915 1729 raft_consensus.cc:1235] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:05.162266 1482 catalog_manager.cc:5526] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:05.318931 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.292351 2651 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:05.109267 2513 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.146996 2520 raft_consensus.cc:3049] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:05.158743 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.164916 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.088785 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.327800 1662 raft_consensus.cc:2463] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:05.316495 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:05.147428 2609 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:05.328207 1628 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:05.328771 1553 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:05.169384 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:05.109875 1583 raft_consensus.cc:2463] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:05.159090 2593 raft_consensus.cc:2743] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:05.190788 2674 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:05.107124 2733 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:05.292537 2644 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:05.329097 1482 catalog_manager.cc:5526] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 2, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.164732 2693 raft_consensus.cc:2743] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:05.153575 1659 raft_consensus.cc:1235] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:05.165743 1580 raft_consensus.cc:1235] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:05.164310 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.171486 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:05.159468 2625 raft_consensus.cc:695] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:05.167609 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.156216 2416 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:05.163215 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.314816 1582 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.348140 1702 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:05.348606 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.361807 1586 raft_consensus.cc:2388] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:05.361981 1702 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:05.330250 2520 raft_consensus.cc:738] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:05.165354 2630 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.163637 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.318396 2197 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:05.353477 2615 raft_consensus.cc:2798] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:05.166209 1730 raft_consensus.cc:2463] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:05.355888 1653 raft_consensus.cc:1235] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:05.356302 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:54:05.356547 2741 log.cc:926] Time spent T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250114 02:54:05.355052 2674 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.357532 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.357928 2644 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.354537 1482 catalog_manager.cc:5526] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.358178 2734 consensus_queue.cc:237] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.363040 1702 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:05.161998 2730 raft_consensus.cc:2743] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:05.363351 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:05.349510 2513 raft_consensus.cc:2743] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:05.351691 1628 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:05.327693 2512 consensus_queue.cc:1046] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20250114 02:54:05.353313 1581 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.368598 1584 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:05.331416 2651 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.162503 2710 consensus_queue.cc:237] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.331358 2688 consensus_queue.cc:1046] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20250114 02:54:05.378266 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.373118 2674 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.332181 2724 consensus_queue.cc:1035] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20250114 02:54:05.379076 1583 raft_consensus.cc:2463] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:05.368100 2615 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:05.367867 2197 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:05.381711 1587 raft_consensus.cc:2388] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:05.388726 2629 raft_consensus.cc:2798] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:05.352079 2609 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.365141 2743 raft_consensus.cc:2743] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:05.383565 1482 catalog_manager.cc:5526] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 2, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.384740 1628 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:05.380522 2625 consensus_queue.cc:237] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.369671 2742 raft_consensus.cc:2798] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:05.368374 2732 consensus_queue.cc:237] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.406927 2615 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:05.367355 1553 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:05.398502 1585 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:05.382304 2416 raft_consensus.cc:2743] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:05.403848 1586 raft_consensus.cc:1270] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 02:54:05.419417 1628 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:05.404713 1702 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:05.415145 1655 raft_consensus.cc:2405] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:05.421027 1586 raft_consensus.cc:1270] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:54:05.417940 1482 catalog_manager.cc:5526] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
W20250114 02:54:05.420079 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 52777.7us
I20250114 02:54:05.424382 1553 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:05.408488 2197 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:05.426088 1482 catalog_manager.cc:5526] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.420282 2688 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:05.428071 1702 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:05.422505 2513 raft_consensus.cc:2743] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:05.415189 1583 raft_consensus.cc:1235] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:05.427507 1553 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:05.419930 2629 raft_consensus.cc:695] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:05.432914 1482 catalog_manager.cc:5526] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.434610 1702 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:05.439808 1735 raft_consensus.cc:1270] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250114 02:54:05.440178 1735 pending_rounds.cc:77] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 2
I20250114 02:54:05.440382 1735 pending_rounds.cc:99] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250114 02:54:05.439746 1482 catalog_manager.cc:5526] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 3, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.440608 1735 raft_consensus.cc:2883] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:05.426738 2643 raft_consensus.cc:2798] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:05.429101 2743 raft_consensus.cc:2743] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:05.444339 1482 catalog_manager.cc:5526] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 2, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.454043 2735 consensus_queue.cc:1035] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20250114 02:54:05.456004 1657 raft_consensus.cc:1270] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:54:05.463512 2611 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.464083 2653 raft_consensus.cc:3049] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:05.465111 2653 raft_consensus.cc:738] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:05.465673 2615 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.466784 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.388339 2520 consensus_queue.cc:260] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.465893 2653 consensus_queue.cc:260] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.454077 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.477078 1731 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:05.475615 1482 catalog_manager.cc:5526] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.444710 1584 raft_consensus.cc:1235] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:05.467376 1585 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:05.478400 1482 catalog_manager.cc:5526] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 2, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.489878 1662 raft_consensus.cc:1270] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:54:05.445705 2508 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250114 02:54:05.456661 2744 log.cc:926] Time spent T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250114 02:54:05.490099 1482 catalog_manager.cc:5526] T 5e00ae680c2d4c5cbb5154723228286f P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.503554 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.445989 2442 raft_consensus.cc:2798] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:05.445852 2688 raft_consensus.cc:695] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:05.498170 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.502869 1482 catalog_manager.cc:5526] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.445853 2647 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:05.504868 1735 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:05.476213 1733 raft_consensus.cc:1270] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250114 02:54:05.508666 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.523138 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:05.456771 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.510771 1482 catalog_manager.cc:5526] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.523537 1734 raft_consensus.cc:2371] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4. Re-sending same reply.
I20250114 02:54:05.512542 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.529155 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.530030 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.530580 1735 raft_consensus.cc:2371] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3. Re-sending same reply.
I20250114 02:54:05.531327 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.532182 1735 raft_consensus.cc:2388] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:05.457795 2528 raft_consensus.cc:3049] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:05.539261 1734 raft_consensus.cc:2388] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:05.513401 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.514181 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.515231 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.456866 2609 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.505445 2442 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:05.504523 2630 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:05.456791 2690 consensus_queue.cc:1035] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20250114 02:54:05.523888 1585 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.492808 2719 raft_consensus.cc:2743] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:05.538913 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.525029 1628 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:05.492884 2508 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:05.474941 2643 raft_consensus.cc:695] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:05.491972 2512 consensus_queue.cc:1035] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20250114 02:54:05.526327 2653 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:05.592720 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:05.492599 2416 consensus_queue.cc:1046] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20250114 02:54:05.522814 1733 pending_rounds.cc:77] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 1
I20250114 02:54:05.528946 1482 catalog_manager.cc:5526] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 2, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
W20250114 02:54:05.594918 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 69994.3us
I20250114 02:54:05.592536 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.594973 1482 catalog_manager.cc:5526] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 3, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 3 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:05.595649 1628 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:05.596609 1727 raft_consensus.cc:2388] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:05.596686 1736 raft_consensus.cc:1270] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250114 02:54:05.597020 1736 pending_rounds.cc:77] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 1
I20250114 02:54:05.597214 1736 pending_rounds.cc:99] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 02:54:05.597607 1736 raft_consensus.cc:2883] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:05.541833 1728 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.502784 2743 consensus_queue.cc:1035] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.089s
I20250114 02:54:05.601817 1482 catalog_manager.cc:5526] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 3, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.575696 1654 raft_consensus.cc:2388] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:05.578419 2644 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.605785 1482 catalog_manager.cc:5526] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 2, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.539652 2651 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.585292 2442 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.592283 2508 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:05.574873 1653 raft_consensus.cc:1235] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:05.577838 1583 raft_consensus.cc:1235] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:05.608853 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.539491 2528 raft_consensus.cc:738] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:05.615897 1727 raft_consensus.cc:2388] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:05.608958 1628 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
W20250114 02:54:05.500636 2674 consensus_queue.cc:1159] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b89550620e1d455d887be72232bb5454" responder_term: 2 status { last_received { term: 2 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 2 index: 2 } }
I20250114 02:54:05.580017 2593 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:05.541857 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.575553 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:05.610101 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.595573 1733 pending_rounds.cc:99] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 02:54:05.530848 2642 consensus_queue.cc:1046] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20250114 02:54:05.574496 2735 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:05.570909 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.620493 1585 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:05.612167 1482 catalog_manager.cc:5526] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 2, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.607182 2646 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:05.604367 2617 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:05.625483 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 45542.6us
I20250114 02:54:05.653357 1482 catalog_manager.cc:5526] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 2, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.607439 2650 raft_consensus.cc:2798] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:05.586218 2385 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:05.636960 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.599576 2630 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.542511 2629 consensus_queue.cc:237] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.582003 2733 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:05.660475 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:54:05.658811 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.628911 1733 raft_consensus.cc:2883] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:05.668967 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.669559 1585 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:05.613950 2644 consensus_queue.cc:1046] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20250114 02:54:05.677435 1553 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:05.662222 1482 catalog_manager.cc:5526] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 0 to 2, leader changed from <none> to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:05.653250 2368 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:05.618733 2651 consensus_queue.cc:1035] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20250114 02:54:05.656786 2650 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:05.681913 1661 raft_consensus.cc:1235] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20250114 02:54:05.681571 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.651223 2609 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:05.529274 2647 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.611212 2688 consensus_queue.cc:237] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.663686 2416 consensus_queue.cc:1225] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer 1e5870f8269b49cf81dc39f1a5c8f5c1 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 02:54:05.693647 2647 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.661586 2617 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.696435 2617 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.688603 2609 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.651154 2735 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.698886 2609 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.699303 2735 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:05.699596 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.699753 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.699993 2735 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:05.700171 1727 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:05.700470 2735 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.688987 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.666059 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:05.715193 2733 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.660779 2615 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.663919 2640 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.715502 2732 consensus_queue.cc:1046] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:05.684751 2690 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:05.717279 1582 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
W20250114 02:54:05.717722 2748 log.cc:926] Time spent T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20250114 02:54:05.667493 2646 raft_consensus.cc:695] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:05.717301 2690 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:05.718976 2690 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:05.719161 2646 consensus_queue.cc:237] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.720945 1656 raft_consensus.cc:2463] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:05.629350 2593 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.690438 1553 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:05.709091 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:05.685353 2611 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.710959 2734 raft_consensus.cc:3049] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:05.685287 2650 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:05.713418 2609 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
W20250114 02:54:05.687139 2747 log.cc:926] Time spent T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20250114 02:54:05.663152 2643 consensus_queue.cc:237] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.715601 1579 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:05.664264 2655 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:05.725165 2735 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:05.725312 2742 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
W20250114 02:54:05.726094 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 71997.8us
W20250114 02:54:05.730680 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 79567.2us
I20250114 02:54:05.681574 2368 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:05.711428 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.731914 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.732769 1587 raft_consensus.cc:1270] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:05.687146 2728 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:05.737746 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 47356.4us
I20250114 02:54:05.739874 1585 raft_consensus.cc:1270] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:05.699400 1583 raft_consensus.cc:1235] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:05.668664 2385 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.731647 2734 raft_consensus.cc:738] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:05.739369 2732 consensus_queue.cc:1225] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer 1e5870f8269b49cf81dc39f1a5c8f5c1 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 02:54:05.679204 2633 consensus_queue.cc:1035] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20250114 02:54:05.734442 2730 raft_consensus.cc:2743] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:05.735461 1580 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.733647 1582 raft_consensus.cc:1270] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:54:05.749114 2742 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.769649 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.764371 1553 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:05.764394 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.675663 2528 consensus_queue.cc:260] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.741194 2630 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.739303 2625 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:05.762539 1702 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:05.762617 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.760267 2615 raft_consensus.cc:491] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:05.760967 2368 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:05.774477 1628 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:05.763490 2756 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:05.767191 1584 raft_consensus.cc:1270] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:05.779139 2615 raft_consensus.cc:513] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.782410 2197 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.783959 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.766197 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:05.785748 1662 raft_consensus.cc:2463] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:05.743893 2609 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.786470 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:05.766819 1728 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:05.747048 2655 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.787364 2633 consensus_queue.cc:1035] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:05.787853 1658 raft_consensus.cc:2388] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:05.772791 2693 raft_consensus.cc:2743] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:05.765970 2710 consensus_queue.cc:1046] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20250114 02:54:05.790804 1702 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:05.788964 2197 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.771435 2733 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:05.744868 2442 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.766674 2513 consensus_queue.cc:1035] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20250114 02:54:05.802440 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.803464 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.804175 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:05.804844 1656 raft_consensus.cc:2463] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:05.806093 1656 raft_consensus.cc:1270] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:05.798267 2734 consensus_queue.cc:260] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.804131 2368 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.808012 2734 raft_consensus.cc:3054] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.808691 1653 raft_consensus.cc:2363] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
I20250114 02:54:05.810027 2368 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.765796 2728 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.811583 2655 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.811939 2728 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.813778 2593 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.814306 2733 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.816370 2520 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:05.816536 2733 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.783138 2743 consensus_queue.cc:1046] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20250114 02:54:05.819661 1659 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:05.824843 2385 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.829535 1581 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:05.830262 2724 raft_consensus.cc:2743] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:05.831089 1586 raft_consensus.cc:1235] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:05.768339 2735 consensus_queue.cc:1046] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250114 02:54:05.832031 2735 consensus_queue.cc:1225] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer 1e5870f8269b49cf81dc39f1a5c8f5c1 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 02:54:05.832540 2710 raft_consensus.cc:3049] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:05.832875 2710 raft_consensus.cc:738] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:05.833359 2508 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.833676 2710 consensus_queue.cc:260] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.834790 2710 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.834945 2615 leader_election.cc:290] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:05.837049 2761 log.cc:926] Time spent T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20250114 02:54:05.838428 2690 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.782941 2756 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.851351 2756 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.810463 1579 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:05.776919 2759 raft_consensus.cc:2798] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:05.855717 2759 raft_consensus.cc:695] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:05.856793 2759 consensus_queue.cc:237] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.840425 2442 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:05.841028 1730 raft_consensus.cc:2388] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:05.865800 2720 raft_consensus.cc:3049] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:05.866245 2720 raft_consensus.cc:738] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:05.867188 2720 consensus_queue.cc:260] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.869182 2720 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:05.873778 1733 raft_consensus.cc:1270] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250114 02:54:05.876839 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 86136.7us
I20250114 02:54:05.881697 1702 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:05.775774 2625 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.882980 2743 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:05.896782 2743 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.886186 2645 raft_consensus.cc:2798] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:05.886626 2647 raft_consensus.cc:3049] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:05.899260 2647 raft_consensus.cc:738] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:05.900388 2647 consensus_queue.cc:260] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.901757 2645 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:05.902268 2645 raft_consensus.cc:3054] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:05.904520 2528 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:05.905860 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:05.908666 2743 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:05.910765 2752 log.cc:926] Time spent T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.236s user 0.003s sys 0.000s
I20250114 02:54:05.912355 2625 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.890668 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:05.896028 2442 consensus_queue.cc:1035] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:05.913655 1653 raft_consensus.cc:1270] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:54:05.916334 2508 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:05.887358 2609 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.918032 2650 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.919250 1657 raft_consensus.cc:1235] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20250114 02:54:05.919659 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 123875us
I20250114 02:54:05.941913 1660 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:05.943938 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.786082 2629 raft_consensus.cc:2798] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:05.950204 1734 raft_consensus.cc:2388] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:05.951274 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.952001 1734 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:05.953351 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.929044 2197 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:05.955129 2197 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:05.930995 2690 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.934890 1654 raft_consensus.cc:2388] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:05.935185 2647 raft_consensus.cc:3054] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:05.922677 1702 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:05.967141 2197 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:05.956542 1734 raft_consensus.cc:2463] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:05.958305 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.959537 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.883877 2710 raft_consensus.cc:2798] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:05.970633 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:05.971354 1587 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:05.983747 1586 raft_consensus.cc:1270] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 02:54:05.986446 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:05.920619 1580 raft_consensus.cc:2463] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:05.987828 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.995677 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.986657 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:05.993685 1587 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:06.003036 1702 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:06.004945 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:05.965974 2629 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:06.006301 2629 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:06.006357 2528 raft_consensus.cc:2743] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:06.008478 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:06.010114 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.011049 1585 raft_consensus.cc:2388] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 5: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:06.009102 1734 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:05.983222 2710 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:06.009486 2719 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:06.032919 2719 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.035024 2719 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:06.009181 2743 consensus_queue.cc:1035] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20250114 02:54:06.037464 2650 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:06.008858 2609 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:06.038959 2609 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.040827 2609 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:06.042415 2760 log.cc:926] Time spent T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.160s user 0.000s sys 0.000s
I20250114 02:54:06.043704 2197 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:06.044349 2197 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.046036 1628 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:06.047204 1628 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.053748 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:06.054469 1586 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 5
I20250114 02:54:06.057416 2197 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:06.031114 2753 log.cc:926] Time spent T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250114 02:54:06.057987 2655 raft_consensus.cc:2743] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:06.058234 1659 raft_consensus.cc:1270] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:06.060017 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.029898 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:06.066979 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:06.063809 2756 raft_consensus.cc:2798] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:06.065140 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:06.006649 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.007098 2653 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:06.057663 1653 raft_consensus.cc:1270] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:06.063882 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:06.071923 1628 leader_election.cc:304] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:05.997511 2732 consensus_queue.cc:1046] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20250114 02:54:06.047029 2710 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:06.094416 1660 raft_consensus.cc:1270] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:06.084434 1730 raft_consensus.cc:3054] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
W20250114 02:54:06.102736 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 331957us
I20250114 02:54:06.091475 2756 raft_consensus.cc:695] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:06.107676 2756 consensus_queue.cc:237] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.094084 2655 raft_consensus.cc:2743] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:06.093333 1733 raft_consensus.cc:1270] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250114 02:54:06.091249 2508 raft_consensus.cc:2743] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:06.111124 1654 raft_consensus.cc:2388] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:06.129781 2733 consensus_queue.cc:1035] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:06.093181 2732 consensus_queue.cc:1225] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer 1e5870f8269b49cf81dc39f1a5c8f5c1 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 02:54:06.126675 1733 pending_rounds.cc:77] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 1
I20250114 02:54:06.136391 1733 pending_rounds.cc:99] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 02:54:06.136688 1733 raft_consensus.cc:2883] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:06.112601 2640 consensus_queue.cc:1035] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20250114 02:54:06.138760 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.140082 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:06.140872 1584 raft_consensus.cc:2388] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:06.142390 1702 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:06.142827 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.143666 2640 raft_consensus.cc:2743] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:06.144448 1580 raft_consensus.cc:1270] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:54:06.146934 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:06.146829 2640 consensus_queue.cc:1035] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:06.147789 1728 raft_consensus.cc:2388] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:06.103858 2759 consensus_queue.cc:1035] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20250114 02:54:06.148604 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:06.149227 1728 raft_consensus.cc:2463] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:54:06.150099 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:54:06.150650 2762 log.cc:926] Time spent T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20250114 02:54:06.124504 2368 consensus_queue.cc:1035] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250114 02:54:06.081601 2653 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.165205 1586 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:06.147636 1730 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:06.173565 1628 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:06.179188 1628 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:06.101398 2734 consensus_queue.cc:1035] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20250114 02:54:06.138274 2629 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.182544 1628 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:06.196911 1553 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.171396 2732 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:06.203222 1553 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.182852 2640 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:06.207738 1553 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.191609 2655 raft_consensus.cc:2798] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:06.196720 2650 raft_consensus.cc:2798] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:06.203917 2688 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:06.202452 1628 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:06.224762 2385 raft_consensus.cc:2743] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:06.238412 1553 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.176924 2645 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.174470 2643 consensus_queue.cc:1035] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.057s
I20250114 02:54:06.270680 2650 raft_consensus.cc:695] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:06.270404 2629 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:06.265105 1584 raft_consensus.cc:1270] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:06.201666 2368 consensus_queue.cc:1035] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20250114 02:54:06.265250 1553 leader_election.cc:304] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.225445 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:06.276422 1553 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.269181 2513 raft_consensus.cc:2743] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:06.277926 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:06.275297 2688 raft_consensus.cc:695] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:06.279479 2688 consensus_queue.cc:237] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.273312 2655 raft_consensus.cc:695] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:06.263088 2640 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.269718 2646 raft_consensus.cc:2798] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
W20250114 02:54:06.264447 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 62152.1us
I20250114 02:54:06.229393 2754 raft_consensus.cc:2743] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:06.275278 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:06.239275 2693 raft_consensus.cc:2743] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:06.208256 2732 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.271497 2645 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:06.290813 1728 raft_consensus.cc:2388] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:06.290843 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:06.288350 2385 raft_consensus.cc:2743] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:06.290776 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:06.300711 1628 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:06.294391 2646 raft_consensus.cc:695] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:06.312835 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.313005 1657 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:06.313867 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:06.316340 2625 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:06.318178 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:06.284437 2640 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:06.291069 2719 raft_consensus.cc:2743] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:06.283586 2655 consensus_queue.cc:237] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.323733 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.271584 2650 consensus_queue.cc:237] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.319546 1730 raft_consensus.cc:2388] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:06.328881 2385 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:06.329322 1659 raft_consensus.cc:2463] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:06.329600 2653 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:06.329341 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:06.351390 1628 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:06.353353 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.330399 2646 consensus_queue.cc:237] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.312867 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:06.334645 1702 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:06.295006 2513 consensus_queue.cc:1035] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20250114 02:54:06.337630 1587 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:06.341979 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:06.342267 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:06.346266 2774 log.cc:926] Time spent T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20250114 02:54:06.336050 1729 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:06.333098 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:06.347016 2729 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:06.362216 2732 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:06.358863 1628 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:06.330535 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.367822 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.362823 2368 consensus_queue.cc:1035] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
W20250114 02:54:06.366426 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 72722.7us
I20250114 02:54:06.364552 2645 raft_consensus.cc:2798] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:06.337193 2625 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.331936 2721 consensus_queue.cc:1035] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20250114 02:54:06.372984 1659 raft_consensus.cc:2463] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:06.445545 2653 raft_consensus.cc:2798] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250114 02:54:06.446108 2742 consensus_meta.cc:287] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.669s user 0.008s sys 0.001s
W20250114 02:54:06.445303 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 86523.1us
I20250114 02:54:06.365437 2629 raft_consensus.cc:2798] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:06.366016 1728 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:06.453168 1587 raft_consensus.cc:1270] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:06.450037 2710 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.457046 1729 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:06.457208 1553 leader_election.cc:400] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
I20250114 02:54:06.458099 1702 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:06.463754 1553 leader_election.cc:403] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250114 02:54:06.451108 2645 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:06.465349 2645 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:06.465530 2593 raft_consensus.cc:2743] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
I20250114 02:54:06.467705 1732 raft_consensus.cc:2388] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:06.468542 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:06.469600 1729 raft_consensus.cc:2405] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:06.452680 1579 raft_consensus.cc:2463] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:06.459156 2528 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:06.471028 2528 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.454102 2640 raft_consensus.cc:491] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:06.473469 2528 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:06.473510 2640 raft_consensus.cc:513] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.475764 2640 leader_election.cc:290] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:06.456048 1628 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.441957 2729 raft_consensus.cc:695] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:06.358404 2385 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.479022 1628 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.487644 2625 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:06.456794 2629 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:06.461314 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:06.503127 2629 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:06.490463 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:06.505450 1656 raft_consensus.cc:2388] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:06.491256 2721 raft_consensus.cc:2743] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:06.459797 2742 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.506945 2729 consensus_queue.cc:237] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.508069 2742 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:06.456877 2653 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:06.508641 2653 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:06.494802 1628 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:06.470969 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:06.495676 2640 consensus_queue.cc:1035] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:06.458546 1657 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:06.511142 1628 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.492755 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:06.518319 1585 raft_consensus.cc:3054] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:06.540057 1628 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:06.541929 1628 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.494948 2650 raft_consensus.cc:2743] T c39189f5c30e4eec95701fe6a370896e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:06.493382 2368 raft_consensus.cc:2743] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:06.617156 1702 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:06.617179 2742 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:06.618958 2732 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:06.619216 2788 log.cc:926] Time spent T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20250114 02:54:06.619256 2742 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.619669 2732 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.621435 2742 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:06.622113 2732 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:06.623044 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:06.621937 2742 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:06.628098 2777 log.cc:926] Time spent T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.366s user 0.001s sys 0.003s
I20250114 02:54:06.631023 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.632019 2385 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:06.632930 2688 raft_consensus.cc:2798] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250114 02:54:06.636087 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 94260.3us
I20250114 02:54:06.642290 1628 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.637727 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:06.652473 1585 raft_consensus.cc:1270] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:54:06.653879 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.654917 1585 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:06.658846 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.660274 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:06.661211 1585 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:06.670820 1585 raft_consensus.cc:2463] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:06.672051 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.672864 1585 raft_consensus.cc:2463] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:06.673857 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.675215 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:06.678963 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.679651 1585 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:06.632818 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
W20250114 02:54:06.682482 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 216530us
I20250114 02:54:06.631028 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.615731 2625 raft_consensus.cc:2743] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:06.650476 2633 raft_consensus.cc:2743] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:06.691040 1553 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:06.641623 2688 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:06.651005 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.656538 2643 raft_consensus.cc:2743] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:06.690196 1659 raft_consensus.cc:2363] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5.
I20250114 02:54:06.692618 2647 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:06.640985 2742 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.693334 1655 raft_consensus.cc:1235] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20250114 02:54:06.696482 2721 raft_consensus.cc:2743] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:06.714520 1553 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:06.709327 1580 raft_consensus.cc:2363] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5.
I20250114 02:54:06.716281 1553 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:06.697569 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:06.721485 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 82091.9us
I20250114 02:54:06.700521 2710 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:06.702853 2688 raft_consensus.cc:3054] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:06.706802 2513 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:06.716883 1579 raft_consensus.cc:2388] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:06.706151 1584 raft_consensus.cc:2463] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:06.797223 1702 leader_election.cc:400] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5.
I20250114 02:54:06.797770 1702 leader_election.cc:403] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20250114 02:54:06.802016 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:54:06.783138 2693 raft_consensus.cc:2798] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:06.729755 2786 raft_consensus.cc:2798] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:06.812147 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:54:06.832667 2787 log.cc:926] Time spent T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20250114 02:54:06.847308 2790 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:06.847805 2793 log.cc:926] Time spent T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250114 02:54:06.847873 2790 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.849970 2790 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:06.794751 2775 raft_consensus.cc:2798] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:06.907007 1628 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
W20250114 02:54:06.863003 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 134061us
I20250114 02:54:06.788013 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:06.716511 2647 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.909224 1628 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:06.907873 1553 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.793751 2710 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:06.731786 2645 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.912662 1628 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:06.923828 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:06.925037 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:06.925654 1734 raft_consensus.cc:3054] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:06.908963 1728 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:06.927958 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:06.928341 1628 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:06.928565 1728 raft_consensus.cc:2463] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:06.929298 2650 raft_consensus.cc:2798] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:06.929510 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:06.929695 2650 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:06.930136 2650 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:06.930747 2633 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:06.931128 2633 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:06.931474 2633 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:06.931385 1628 leader_election.cc:304] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.932475 2728 raft_consensus.cc:2743] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:06.933899 1628 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:06.934790 2728 raft_consensus.cc:2743] T 3a7e023d568848b9861c2976a138d461 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:06.920739 1583 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:06.813337 2693 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:06.836467 2786 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:06.940668 1553 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:06.795531 2513 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.945947 2513 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:06.946447 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.909523 2646 raft_consensus.cc:2743] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:06.968068 1734 raft_consensus.cc:2463] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:06.911082 2710 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:06.973165 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:06.947053 1728 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:06.977373 1734 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:06.979477 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:06.980640 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:06.982163 1734 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
W20250114 02:54:06.949599 2781 log.cc:926] Time spent T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20250114 02:54:06.938202 2643 raft_consensus.cc:2798] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:06.923451 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:06.941043 2790 raft_consensus.cc:2743] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:06.939153 2775 raft_consensus.cc:695] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:06.947512 2693 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:06.939597 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:06.968487 2799 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:06.951061 2786 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:07.006139 2808 consensus_queue.cc:1035] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20250114 02:54:07.006893 2792 log.cc:926] Time spent T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.291s user 0.001s sys 0.000s
I20250114 02:54:07.007633 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 586) took 1066 ms. Trace:
W20250114 02:54:07.009807 2791 log.cc:926] Time spent T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.218s user 0.000s sys 0.001s
W20250114 02:54:07.010968 2798 log.cc:926] Time spent T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
W20250114 02:54:07.012382 2794 log.cc:926] Time spent T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20250114 02:54:07.016290 2197 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:07.018198 2643 raft_consensus.cc:695] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:07.043946 2197 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.001413 2801 raft_consensus.cc:2743] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5.
I20250114 02:54:07.020605 1580 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:07.019673 2742 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625)
I20250114 02:54:07.017767 2790 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:07.031236 2513 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:07.057747 2197 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:07.034874 1733 rpcz_store.cc:276] 0114 02:54:05.941563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:05.946897 (+ 5334us) service_pool.cc:225] Handling call
0114 02:54:06.006134 (+ 59237us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:06.137084 (+130950us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:07.007318 (+870234us) spinlock_profiling.cc:243] Waited 62.2 ms on lock 0x7b540009ced4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:07.007545 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":143001856}
I20250114 02:54:07.062523 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 597) took 1098 ms. Trace:
I20250114 02:54:07.043565 2743 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:07.063159 2782 log.cc:926] Time spent T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.611s user 0.003s sys 0.007s
I20250114 02:54:07.094873 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 595) took 1150 ms. Trace:
I20250114 02:54:07.095288 1729 rpcz_store.cc:276] 0114 02:54:05.944854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:06.478156 (+533302us) service_pool.cc:225] Handling call
0114 02:54:06.478498 (+ 342us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:06.478909 (+ 411us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:06.478963 (+ 54us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:06.479429 (+ 466us) log.cc:844] Serialized 26 byte log entry
0114 02:54:06.580845 (+101416us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:06.580902 (+ 57us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:06.581020 (+ 118us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:07.094657 (+513637us) raft_consensus.cc:1680] finished
0114 02:54:07.094678 (+ 21us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:07.094822 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8164,"raft.queue_time_us":3285,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"thread_start_us":101076,"threads_started":1,"wal-append.queue_time_us":227467}
W20250114 02:54:07.098656 2784 log.cc:926] Time spent T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.701s user 0.015s sys 0.028s
I20250114 02:54:07.101066 2693 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.105443 1733 raft_consensus.cc:1235] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250114 02:54:07.101073 2633 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.107564 1580 raft_consensus.cc:1270] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 02:54:07.107707 2512 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:07.108243 2512 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.108721 2633 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:07.114660 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 174083us
I20250114 02:54:07.115746 1553 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
W20250114 02:54:07.117197 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 318574us
I20250114 02:54:07.117977 1702 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:07.110105 2512 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:07.017166 2799 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.075253 2513 raft_consensus.cc:695] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:07.097754 2742 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:07.124455 1702 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:07.068779 2790 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.133570 1702 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
W20250114 02:54:07.127319 2629 consensus_meta.cc:287] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.622s user 0.007s sys 0.003s
I20250114 02:54:07.068935 2775 consensus_queue.cc:237] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.156935 1583 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:07.140808 1702 leader_election.cc:400] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5.
I20250114 02:54:07.158411 1702 leader_election.cc:304] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:07.159734 1702 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:07.125254 2647 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:07.175801 1583 raft_consensus.cc:1270] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:07.177482 1583 raft_consensus.cc:1270] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:07.074546 2743 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.106422 2643 consensus_queue.cc:237] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.130131 2645 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:07.147763 2735 raft_consensus.cc:2743] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:54:07.147621 2653 consensus_meta.cc:287] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.639s user 0.010s sys 0.000s
I20250114 02:54:07.131078 2804 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:07.150722 2710 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:07.136684 2512 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250114 02:54:07.201840 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 154305us
I20250114 02:54:07.171564 2730 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:07.140079 2816 raft_consensus.cc:2743] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:07.129966 2742 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.192584 2806 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:07.191419 2693 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:07.161198 2629 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.192695 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48636 (request call id 411) took 1049 ms. Trace:
I20250114 02:54:07.211812 1628 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:07.151909 2650 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.204481 2512 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:07.200996 2809 raft_consensus.cc:2798] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:07.214164 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 437) took 1027 ms. Trace:
I20250114 02:54:07.201833 2653 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.151070 2513 consensus_queue.cc:237] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:07.257118 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 45379.2us
I20250114 02:54:07.258275 2650 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:07.207763 2442 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:07.215055 2728 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:07.204171 2804 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:07.282742 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 83917.6us
I20250114 02:54:07.544327 1628 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:07.067436 1730 rpcz_store.cc:276] 0114 02:54:05.963795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:06.503615 (+539820us) service_pool.cc:225] Handling call
0114 02:54:06.503925 (+ 310us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:06.504308 (+ 383us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:06.504342 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:06.504741 (+ 399us) log.cc:844] Serialized 26 byte log entry
0114 02:54:06.690116 (+185375us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:06.692369 (+ 2253us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:06.698099 (+ 5730us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:07.062304 (+364205us) raft_consensus.cc:1680] finished
0114 02:54:07.062329 (+ 25us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:07.062464 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2053,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"thread_start_us":173622,"threads_started":1,"wal-append.queue_time_us":289411}
I20250114 02:54:07.210740 2735 raft_consensus.cc:3049] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:07.260692 2816 raft_consensus.cc:491] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:07.211212 2710 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:07.567728 1628 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:07.568745 2197 consensus_queue.cc:1035] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:07.570583 2197 raft_consensus.cc:2798] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:07.571099 2197 raft_consensus.cc:695] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
W20250114 02:54:07.360613 2766 log.cc:926] Time spent T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 1.009s user 0.002s sys 0.001s
I20250114 02:54:07.257058 2385 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:07.571978 2197 consensus_queue.cc:237] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.217031 1658 raft_consensus.cc:1270] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 02:54:07.255152 2609 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:07.450361 2799 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:07.323297 2809 raft_consensus.cc:491] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:07.323531 2512 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:07.322201 2812 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:07.275681 2647 raft_consensus.cc:2798] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:07.576373 2743 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:07.593379 2805 log.cc:926] Time spent T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.394s user 0.001s sys 0.000s
I20250114 02:54:07.556604 2735 raft_consensus.cc:738] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:07.555725 2655 raft_consensus.cc:2798] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:07.461488 2790 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:07.212119 2730 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:07.585341 2688 consensus_meta.cc:287] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.786s user 0.011s sys 0.001s
I20250114 02:54:07.593899 2800 consensus_queue.cc:1035] T 0ce7223c99df448483de68f3463ba115 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
W20250114 02:54:07.623634 2821 log.cc:926] Time spent T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
W20250114 02:54:07.616700 2786 consensus_meta.cc:287] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.582s user 0.007s sys 0.001s
I20250114 02:54:07.553628 2728 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.618521 2647 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:07.615097 2809 raft_consensus.cc:3054] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:07.618271 2799 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:07.578135 2710 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.619577 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:07.588039 2385 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.624250 2814 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:07.635061 2655 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:07.318775 1733 rpcz_store.cc:276] 0114 02:54:06.186954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:07.106393 (+919439us) service_pool.cc:225] Handling call
0114 02:54:07.137928 (+ 31535us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:07.180066 (+ 42138us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:07.187341 (+ 7275us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:07.192587 (+ 5246us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:07.194893 (+ 2306us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:07.206407 (+ 11514us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:07.212955 (+ 6548us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:07.625429 2742 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:07.678063 2813 log.cc:926] Time spent T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.402s user 0.001s sys 0.000s
I20250114 02:54:07.254896 2806 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.698309 2810 raft_consensus.cc:2743] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:07.576215 2816 raft_consensus.cc:513] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:07.702455 2823 log.cc:926] Time spent T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
I20250114 02:54:07.702844 2811 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:07.703286 2822 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:07.707619 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 422) took 2165 ms. Trace:
I20250114 02:54:07.708782 2655 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:07.594275 2609 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:07.718573 2817 log.cc:926] Time spent T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.250s user 0.000s sys 0.001s
I20250114 02:54:07.701339 1730 raft_consensus.cc:2463] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:07.685050 2629 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:07.641471 2786 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.733047 1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48636 (request call id 409) took 1683 ms. Trace:
I20250114 02:54:07.742883 1653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48636 (request call id 412) took 1592 ms. Trace:
I20250114 02:54:07.744407 2593 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:07.682902 2647 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:07.749781 1736 rpcz_store.cc:276] 0114 02:54:05.542468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:05.627877 (+ 85409us) service_pool.cc:225] Handling call
0114 02:54:05.658981 (+ 31104us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:05.691547 (+ 32566us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:05.691589 (+ 42us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:05.721917 (+ 30328us) log.cc:844] Serialized 26 byte log entry
0114 02:54:06.081078 (+359161us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:06.234757 (+153679us) log.cc:844] Serialized 14 byte log entry
0114 02:54:06.236146 (+ 1389us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:06.236225 (+ 79us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:07.026634 (+790409us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b48001e0258. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbceaf99581 00007fbceaf90677 00007fbceaf9050a 00007fbceb0849bf 00007fbceb084687 00007fbceb098b8c 00007fbceb098b38 00007fbceb098ab8 00007fbceb098a73 00007fbceb097d1c 00007fbcea004d16
0114 02:54:07.097523 (+ 70889us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b440002494c. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce30ad7c5 00007fbce30ae1e2 00007fbce5d63da1 00007fbce5d5f68a 00007fbce5d534ae 00007fbce5d527f6 00007fbce5d52418 00007fbce5d5bdf1 00007fbce5d5bda9 00007fbce5d5bd39
0114 02:54:07.707426 (+609903us) raft_consensus.cc:1680] finished
0114 02:54:07.707448 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:07.707564 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67302,"prepare.queue_time_us":47342,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":34486,"raft.run_cpu_time_us":3360,"raft.run_wall_time_us":798567,"spinlock_wait_cycles":75742464,"thread_start_us":457737,"threads_started":2,"wal-append.queue_time_us":463230}
I20250114 02:54:07.640996 2688 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.720482 2804 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:07.761207 2770 log.cc:926] Time spent T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 1.426s user 0.001s sys 0.003s
I20250114 02:54:07.765085 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:07.769045 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 560) took 1826 ms. Trace:
I20250114 02:54:07.696728 2653 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:07.754498 2734 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:07.782193 2385 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:07.783389 1732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 591) took 1839 ms (client timeout 1713 ms). Trace:
I20250114 02:54:07.783998 1729 raft_consensus.cc:1235] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:07.784188 2728 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:07.785115 1659 raft_consensus.cc:1235] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:07.786671 2816 leader_election.cc:290] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:07.725968 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:07.932860 1658 raft_consensus.cc:2463] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:07.934021 1578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 405) took 2463 ms. Trace:
I20250114 02:54:07.781651 2822 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.778533 2811 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.636018 2812 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.797895 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:07.839793 2688 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:07.844651 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:07.964578 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 439) took 1771 ms. Trace:
I20250114 02:54:07.964947 1736 rpcz_store.cc:276] 0114 02:54:06.193437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:07.963742 (+1770305us) service_pool.cc:225] Handling call
0114 02:54:07.964080 (+ 338us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:07.964252 (+ 172us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:07.964281 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:07.964334 (+ 53us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:07.964367 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:07.964427 (+ 60us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:07.964526 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:07.934973 2734 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.968504 2734 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:07.790630 1653 rpcz_store.cc:276] 0114 02:54:06.150231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:06.150409 (+ 178us) service_pool.cc:225] Handling call
0114 02:54:06.150911 (+ 502us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:06.151340 (+ 429us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:06.151406 (+ 66us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:06.151902 (+ 496us) log.cc:844] Serialized 26 byte log entry
0114 02:54:06.202666 (+ 50764us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:06.227374 (+ 24708us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:06.231381 (+ 4007us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:07.741268 (+1509887us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b24001be6a8. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce2ff39d4 00007fbce2ff1adb 00007fbce3080289 00007fbce307ff15 00007fbce307f470 00007fbce5d7ff88 00007fbce5d806e8 00007fbce5d57f98 00007fbce5d533aa 00007fbce5d527f6
0114 02:54:07.742495 (+ 1227us) raft_consensus.cc:1680] finished
0114 02:54:07.742545 (+ 50us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:07.742812 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1307,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"spinlock_wait_cycles":24027904,"thread_start_us":39130,"threads_started":1,"wal-append.queue_time_us":323117}
I20250114 02:54:07.831871 2807 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:07.836179 1579 raft_consensus.cc:1270] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:07.974967 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:07.695459 2799 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.715708 2814 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:07.854619 2828 log.cc:926] Time spent T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20250114 02:54:07.269873 1654 rpcz_store.cc:276] 0114 02:54:06.138886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:06.139066 (+ 180us) service_pool.cc:225] Handling call
0114 02:54:06.139500 (+ 434us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:06.139894 (+ 394us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:06.139936 (+ 42us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:06.140451 (+ 515us) log.cc:844] Serialized 26 byte log entry
0114 02:54:06.209294 (+ 68843us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:06.227277 (+ 17983us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:06.231356 (+ 4079us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:07.062608 (+831252us) spinlock_profiling.cc:243] Waited 1.68 ms on lock 0x7b440002494c. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce30ad7c5 00007fbce30ae1e2 00007fbce5d63da1 00007fbce5d5f68a 00007fbce5d534ae 00007fbce5d527f6 00007fbce5d52418 00007fbce5d5bdf1 00007fbce5d5bda9 00007fbce5d5bd39
0114 02:54:07.136758 (+ 74150us) raft_consensus.cc:1680] finished
0114 02:54:07.136787 (+ 29us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:07.180874 (+ 44087us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2069,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"spinlock_wait_cycles":3873664,"thread_start_us":52765,"threads_started":1,"wal-append.queue_time_us":222719}
I20250114 02:54:07.982864 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:07.984025 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:07.984720 1654 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:07.847748 2593 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.988269 2593 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:07.787863 1661 rpcz_store.cc:276] 0114 02:54:06.049648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:06.052686 (+ 3038us) service_pool.cc:225] Handling call
0114 02:54:06.077329 (+ 24643us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:06.080023 (+ 2694us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:06.080064 (+ 41us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:06.799809 (+719745us) spinlock_profiling.cc:243] Waited 4.44 ms on lock 0x7b2c0000f018. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce4ab6741 00007fbce4ac5d72 00007fbce6111141 00007fbce60d558a 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782
0114 02:54:06.800150 (+ 341us) log.cc:844] Serialized 26 byte log entry
0114 02:54:07.044476 (+244326us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:07.044523 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:07.044612 (+ 89us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:07.722259 (+677647us) raft_consensus.cc:1680] finished
0114 02:54:07.722282 (+ 23us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:07.732952 (+ 10670us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6875,"raft.queue_time_us":412,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":5272,"spinlock_wait_cycles":10223488,"thread_start_us":243695,"threads_started":1,"wal-append.queue_time_us":327008}
I20250114 02:54:07.989351 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:07.990027 1661 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:07.991003 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:07.792241 1728 consensus_meta.cc:287] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.792s user 0.011s sys 0.000s
I20250114 02:54:07.994743 1728 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:07.995846 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 616) took 1315 ms. Trace:
I20250114 02:54:07.996197 1728 rpcz_store.cc:276] 0114 02:54:06.680312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:06.929258 (+248946us) service_pool.cc:225] Handling call
0114 02:54:07.995788 (+1066530us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:07.997856 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 611) took 1318 ms. Trace:
I20250114 02:54:07.859619 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 585) took 1210 ms. Trace:
I20250114 02:54:07.998733 1582 rpcz_store.cc:276] 0114 02:54:06.648967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:07.039747 (+390780us) service_pool.cc:225] Handling call
0114 02:54:07.040103 (+ 356us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:07.040552 (+ 449us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:07.040601 (+ 49us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:07.041120 (+ 519us) log.cc:844] Serialized 26 byte log entry
0114 02:54:07.293028 (+251908us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:07.293093 (+ 65us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:07.293194 (+ 101us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:07.809163 (+515969us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b180038c860. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce319c7e8 00007fbce319fcc1 00007fbce319fc79 00007fbce319fc09 00007fbce319fbd1 00007fbce319eecd 00007fbcec1d9234 00007fbcec1d9069 00007fbce317efe6
0114 02:54:07.859252 (+ 50089us) raft_consensus.cc:1680] finished
0114 02:54:07.859284 (+ 32us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:07.859549 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7857,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"spinlock_wait_cycles":4199808,"thread_start_us":225181,"threads_started":1,"wal-append.queue_time_us":314646}
I20250114 02:54:08.001428 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:08.002127 1582 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:07.861538 2786 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:07.787544 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:07.870874 2609 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:07.866374 1730 raft_consensus.cc:1235] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4. Ops: []
W20250114 02:54:07.872500 1657 consensus_meta.cc:287] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 1.237s user 0.010s sys 0.000s
I20250114 02:54:07.876173 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.002794 2807 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.863070 2730 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:07.878355 2767 log.cc:926] Time spent T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 1.611s user 0.003s sys 0.001s
W20250114 02:54:08.114687 2197 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.019s sys 0.049s
W20250114 02:54:07.883666 2772 log.cc:926] Time spent T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 1.531s user 0.004s sys 0.000s
I20250114 02:54:07.897909 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48636 (request call id 419) took 1405 ms. Trace:
W20250114 02:54:07.900645 2764 log.cc:926] Time spent T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 1.632s user 0.015s sys 0.020s
I20250114 02:54:07.829331 2512 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:07.751194 2732 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
W20250114 02:54:07.913537 1732 rpcz_store.cc:269] 0114 02:54:05.943481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:05.959318 (+ 15837us) service_pool.cc:225] Handling call
0114 02:54:07.783108 (+1823790us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54002add54. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:07.783268 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1133787904}
I20250114 02:54:07.931095 1656 raft_consensus.cc:2363] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
I20250114 02:54:07.931905 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.131117 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:07.933970 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.136746 1582 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:08.136922 1658 raft_consensus.cc:2463] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:08.137665 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48632 (request call id 489) took 1001 ms. Trace:
I20250114 02:54:08.138065 1658 rpcz_store.cc:276] 0114 02:54:07.136300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:07.933686 (+797386us) service_pool.cc:225] Handling call
0114 02:54:08.137592 (+203906us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:07.794124 2806 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:07.914848 1734 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.056s sys 0.070s
I20250114 02:54:08.118621 1655 rpcz_store.cc:276] 0114 02:54:06.492369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:06.494062 (+ 1693us) service_pool.cc:225] Handling call
0114 02:54:06.495597 (+ 1535us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 02:54:06.728478 (+232881us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:07.897606 (+1169128us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b5400017554. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:07.897825 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":325388032}
W20250114 02:54:08.139348 1734 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.935s user 0.056s sys 0.070s
I20250114 02:54:08.140054 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 429) took 2012 ms. Trace:
I20250114 02:54:08.140156 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48632 (request call id 486) took 1007 ms. Trace:
I20250114 02:54:08.140583 1659 rpcz_store.cc:276] 0114 02:54:07.132340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:07.931576 (+799236us) service_pool.cc:225] Handling call
0114 02:54:08.140100 (+208524us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.140379 1734 rpcz_store.cc:276] 0114 02:54:06.127597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:07.003609 (+876012us) service_pool.cc:225] Handling call
0114 02:54:07.003904 (+ 295us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:07.144434 (+140530us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:07.144479 (+ 45us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:07.179818 (+ 35339us) log.cc:844] Serialized 26 byte log entry
0114 02:54:08.139708 (+959890us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:08.139750 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.139816 (+ 66us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:08.139865 (+ 49us) raft_consensus.cc:1680] finished
0114 02:54:08.139876 (+ 11us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.140009 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":60704,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":5874,"thread_start_us":1062561,"threads_started":2,"wal-append.queue_time_us":713200}
I20250114 02:54:07.777436 2809 raft_consensus.cc:513] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.144131 2809 leader_election.cc:290] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:07.905570 1581 rpcz_store.cc:276] 0114 02:54:05.942323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:05.962542 (+ 20219us) service_pool.cc:225] Handling call
0114 02:54:07.561769 (+1599227us) spinlock_profiling.cc:243] Waited 1.39 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:54:07.585784 (+ 24015us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:07.766917 (+181133us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:07.766967 (+ 50us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:07.767032 (+ 65us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:07.767085 (+ 53us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:07.768734 (+ 1649us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b540033b8d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:07.768789 (+ 55us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:07.768947 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":381502592}
I20250114 02:54:07.962595 2812 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:08.139369 2609 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:07.971940 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 608) took 1293 ms. Trace:
I20250114 02:54:07.972496 2822 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:07.975816 1583 raft_consensus.cc:2463] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:08.145620 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:08.145368 2609 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.146692 1581 raft_consensus.cc:2463] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:08.146842 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:08.147787 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:08.148111 2609 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:08.148375 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 596) took 1154 ms. Trace:
I20250114 02:54:08.148731 1583 rpcz_store.cc:276] 0114 02:54:06.994183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.147682 (+1153499us) service_pool.cc:225] Handling call
0114 02:54:08.147933 (+ 251us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.148074 (+ 141us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:08.148101 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:08.148140 (+ 39us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:08.148173 (+ 33us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.148236 (+ 63us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.148326 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.148519 1581 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 2.
I20250114 02:54:08.149969 1583 raft_consensus.cc:3049] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:07.978037 2811 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:07.981994 2814 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:07.715878 2735 consensus_queue.cc:260] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.150866 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 424) took 2575 ms. Trace:
I20250114 02:54:08.151249 2735 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:08.151360 1727 rpcz_store.cc:276] 0114 02:54:05.575515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:05.700772 (+125257us) service_pool.cc:225] Handling call
0114 02:54:05.701120 (+ 348us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 02:54:05.701442 (+ 322us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:05.701471 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 02:54:05.735509 (+ 34038us) log.cc:844] Serialized 52 byte log entry
0114 02:54:06.103342 (+367833us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:06.206845 (+103503us) log.cc:844] Serialized 14 byte log entry
0114 02:54:06.236355 (+ 29510us) log.cc:844] Serialized 14 byte log entry
0114 02:54:06.238179 (+ 1824us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:06.238247 (+ 68us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:07.029332 (+791085us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b18000d1320. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce319c914 00007fbce319fcc1 00007fbce319fc79 00007fbce319fc09 00007fbce319fbd1 00007fbce319eecd 00007fbcec1d9234 00007fbcec1d9069 00007fbce317efe6
0114 02:54:07.801745 (+772413us) spinlock_profiling.cc:243] Waited 8.87 ms on lock 0x7b440002494c. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce30ad7c5 00007fbce30ae1e2 00007fbce5d63da1 00007fbce5d5f68a 00007fbce5d534ae 00007fbce5d527f6 00007fbce5d52418 00007fbce5d5bdf1 00007fbce5d5bda9 00007fbce5d5bd39
0114 02:54:07.803031 (+ 1286us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b18000d1320. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce319c914 00007fbce319fcc1 00007fbce319fc79 00007fbce319fc09 00007fbce319fbd1 00007fbce319eecd 00007fbcec1d9234 00007fbcec1d9069 00007fbce317efe6
0114 02:54:08.150635 (+347604us) raft_consensus.cc:1680] finished
0114 02:54:08.150661 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.150805 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37363,"prepare.queue_time_us":26997,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":3349,"raft.run_cpu_time_us":460,"raft.run_wall_time_us":66651,"spinlock_wait_cycles":33299328,"thread_start_us":418888,"threads_started":2,"wal-append.queue_time_us":444634}
I20250114 02:54:08.005550 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 430) took 1877 ms. Trace:
I20250114 02:54:08.154716 1729 rpcz_store.cc:276] 0114 02:54:06.127969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:07.096866 (+968897us) service_pool.cc:225] Handling call
0114 02:54:07.783803 (+686937us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:07.923649 (+139846us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.005322 (+ 81673us) spinlock_profiling.cc:243] Waited 687 ms on lock 0x7b54002add50. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d4a09 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:08.005484 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1579223296}
I20250114 02:54:08.083490 2688 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:07.957469 2655 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:07.877262 2795 log.cc:926] Time spent T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.909s user 0.001s sys 0.001s
I20250114 02:54:08.104360 2799 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:08.105986 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 583) took 1457 ms. Trace:
I20250114 02:54:08.157023 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 615) took 1477 ms. Trace:
I20250114 02:54:07.883225 2710 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:08.117064 2197 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s user 0.019s sys 0.050s
I20250114 02:54:08.095038 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 438) took 1906 ms. Trace:
I20250114 02:54:08.088032 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 582) took 1438 ms. Trace:
I20250114 02:54:07.998180 1728 rpcz_store.cc:276] 0114 02:54:06.679074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:07.997055 (+1317981us) service_pool.cc:225] Handling call
0114 02:54:07.997348 (+ 293us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:07.997525 (+ 177us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:07.997556 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:07.997597 (+ 41us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:07.997637 (+ 40us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:07.997693 (+ 56us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:07.997807 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.107414 2804 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.123821 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 415) took 2614 ms. Trace:
I20250114 02:54:08.124905 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 419) took 2613 ms. Trace:
I20250114 02:54:08.095829 1733 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:08.109472 2593 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:08.095405 1657 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:08.165721 1735 rpcz_store.cc:276] 0114 02:54:05.509244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:05.542711 (+ 33467us) service_pool.cc:225] Handling call
0114 02:54:05.615020 (+ 72309us) spinlock_profiling.cc:243] Waited 40.4 ms on lock 0x7b5800023d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:54:05.615175 (+ 155us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:05.668773 (+ 53598us) raft_consensus.cc:1529] Early marking committed up to index 1
0114 02:54:05.759762 (+ 90989us) log.cc:844] Serialized 14 byte log entry
0114 02:54:06.155271 (+395509us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:06.155760 (+ 489us) log.cc:844] Serialized 26 byte log entry
0114 02:54:06.155893 (+ 133us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:06.156994 (+ 1101us) log.cc:844] Serialized 14 byte log entry
0114 02:54:06.164897 (+ 7903us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:06.164964 (+ 67us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:06.445877 (+280913us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b18002dd3c0. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce319c914 00007fbce319fcc1 00007fbce319fc79 00007fbce319fc09 00007fbce319fbd1 00007fbce319eecd 00007fbcec1d9234 00007fbcec1d9069 00007fbce317efe6
0114 02:54:07.747548 (+1301671us) spinlock_profiling.cc:243] Waited 3.87 ms on lock 0x7b18002dd3c0. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce319c7e8 00007fbce319fcc1 00007fbce319fc79 00007fbce319fc09 00007fbce319fbd1 00007fbce319eecd 00007fbcec1d9234 00007fbcec1d9069 00007fbce317efe6
0114 02:54:07.798717 (+ 51169us) spinlock_profiling.cc:243] Waited 3.39 ms on lock 0x7b440002494c. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce30ad7c5 00007fbce30ae1e2 00007fbce5d63da1 00007fbce5d5f68a 00007fbce5d534ae 00007fbce5d527f6 00007fbce5d52418 00007fbce5d5bdf1 00007fbce5d5bda9 00007fbce5d5bd39
0114 02:54:08.123590 (+324873us) raft_consensus.cc:1680] finished
0114 02:54:08.123617 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.123747 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98594,"prepare.queue_time_us":108276,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":65,"raft.queue_time_us":61914,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":10296,"spinlock_wait_cycles":115216768,"thread_start_us":453561,"threads_started":2,"wal-append.queue_time_us":465131}
I20250114 02:54:08.172662 1735 raft_consensus.cc:1270] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:08.173344 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 619) took 1492 ms. Trace:
I20250114 02:54:08.173705 1735 rpcz_store.cc:276] 0114 02:54:06.681141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.171897 (+1490756us) service_pool.cc:225] Handling call
0114 02:54:08.172272 (+ 375us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.173061 (+ 789us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.173284 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.168496 1731 rpcz_store.cc:276] 0114 02:54:05.511208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:05.600890 (+ 89682us) service_pool.cc:225] Handling call
0114 02:54:05.610944 (+ 10054us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:05.737487 (+126543us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:05.737531 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:05.744868 (+ 7337us) log.cc:844] Serialized 26 byte log entry
0114 02:54:06.078787 (+333919us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:06.087101 (+ 8314us) log.cc:844] Serialized 14 byte log entry
0114 02:54:06.087348 (+ 247us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:06.087415 (+ 67us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:07.899316 (+1811901us) spinlock_profiling.cc:243] Waited 4.21 ms on lock 0x7b440002494c. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce30ad7c5 00007fbce30ae1e2 00007fbce5d63da1 00007fbce5d5f68a 00007fbce5d534ae 00007fbce5d527f6 00007fbce5d52418 00007fbce5d5bdf1 00007fbce5d5bda9 00007fbce5d5bd39
0114 02:54:08.124557 (+225241us) raft_consensus.cc:1680] finished
0114 02:54:08.124685 (+ 128us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.124841 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80690,"prepare.queue_time_us":26210,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":24,"raft.queue_time_us":88721,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":7162,"spinlock_wait_cycles":9692544,"thread_start_us":355377,"threads_started":2,"wal-append.queue_time_us":443952}
I20250114 02:54:08.145720 1734 raft_consensus.cc:1235] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:08.140533 1732 raft_consensus.cc:1270] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 02:54:08.178097 1732 pending_rounds.cc:77] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 1
I20250114 02:54:08.145330 1736 rpcz_store.cc:276] 0114 02:54:06.678328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:07.965935 (+1287607us) service_pool.cc:225] Handling call
0114 02:54:07.966218 (+ 283us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:07.966412 (+ 194us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:07.966440 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:07.966481 (+ 41us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:07.966507 (+ 26us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:07.971729 (+ 5222us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:07.971877 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.178777 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 598) took 1184 ms. Trace:
I20250114 02:54:08.150349 1583 raft_consensus.cc:738] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:08.160292 1729 raft_consensus.cc:1270] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250114 02:54:08.180594 1729 pending_rounds.cc:77] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 2
I20250114 02:54:08.180878 1729 pending_rounds.cc:99] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250114 02:54:08.181123 1729 raft_consensus.cc:2883] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:08.178323 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 614) took 1498 ms. Trace:
I20250114 02:54:08.181854 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 617) took 1501 ms. Trace:
I20250114 02:54:08.181916 1734 rpcz_store.cc:276] 0114 02:54:06.679812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.142350 (+1462538us) service_pool.cc:225] Handling call
0114 02:54:08.145566 (+ 3216us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.178094 (+ 32528us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.178274 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.182204 1729 rpcz_store.cc:276] 0114 02:54:06.680540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.155630 (+1475090us) service_pool.cc:225] Handling call
0114 02:54:08.155941 (+ 311us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.181651 (+ 25710us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.181809 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.183502 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 625) took 1245 ms. Trace:
I20250114 02:54:08.178355 1732 pending_rounds.cc:99] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 02:54:08.183967 1732 raft_consensus.cc:2883] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:08.183871 1734 rpcz_store.cc:276] 0114 02:54:06.938112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.182792 (+1244680us) service_pool.cc:225] Handling call
0114 02:54:08.183072 (+ 280us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.183227 (+ 155us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:08.183252 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:08.183289 (+ 37us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:08.183317 (+ 28us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.183365 (+ 48us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.183454 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.179330 1736 raft_consensus.cc:1270] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 02:54:08.186480 1733 raft_consensus.cc:1270] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:08.162881 1579 rpcz_store.cc:276] 0114 02:54:06.648009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:06.853213 (+205204us) service_pool.cc:225] Handling call
0114 02:54:07.835547 (+982334us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:54:07.835753 (+ 206us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:07.976673 (+140920us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.086026 (+109353us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28208512}
I20250114 02:54:08.159278 2807 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:08.187120 1657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48632 (request call id 478) took 1862 ms (client timeout 1808 ms). Trace:
I20250114 02:54:07.934535 1578 rpcz_store.cc:276] 0114 02:54:05.472093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:05.719946 (+247853us) service_pool.cc:225] Handling call
0114 02:54:05.720251 (+ 305us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:05.720570 (+ 319us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:05.720602 (+ 32us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:05.721006 (+ 404us) log.cc:844] Serialized 26 byte log entry
0114 02:54:06.106480 (+385474us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:06.118082 (+ 11602us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:06.118477 (+ 395us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:07.758252 (+1639775us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b440002494c. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce30ad7c5 00007fbce30ae1e2 00007fbce5d63da1 00007fbce5d5f68a 00007fbce5d534ae 00007fbce5d527f6 00007fbce5d52418 00007fbce5d5bdf1 00007fbce5d5bda9 00007fbce5d5bd39
0114 02:54:07.933626 (+175374us) raft_consensus.cc:1680] finished
0114 02:54:07.933656 (+ 30us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:07.933933 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":922,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"spinlock_wait_cycles":7576448,"thread_start_us":378707,"threads_started":1,"wal-append.queue_time_us":610901}
W20250114 02:54:08.187738 1657 rpcz_store.cc:269] 0114 02:54:06.315346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:06.332846 (+ 17500us) service_pool.cc:225] Handling call
0114 02:54:08.177438 (+1844592us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.188056 1733 raft_consensus.cc:1270] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:08.188309 1579 raft_consensus.cc:1270] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:08.190132 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 613) took 1510 ms. Trace:
I20250114 02:54:08.190634 1732 rpcz_store.cc:276] 0114 02:54:06.679563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.139892 (+1460329us) service_pool.cc:225] Handling call
0114 02:54:08.140193 (+ 301us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.184397 (+ 44204us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.190062 (+ 5665us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.183504 1729 raft_consensus.cc:1270] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:08.192693 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 626) took 1247 ms. Trace:
I20250114 02:54:08.193027 1729 rpcz_store.cc:276] 0114 02:54:06.945508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.182998 (+1237490us) service_pool.cc:225] Handling call
0114 02:54:08.183228 (+ 230us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.192416 (+ 9188us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.192638 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.160142 1727 rpcz_store.cc:276] 0114 02:54:06.680053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.153979 (+1473926us) service_pool.cc:225] Handling call
0114 02:54:08.154292 (+ 313us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.154463 (+ 171us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:08.154497 (+ 34us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:08.154544 (+ 47us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:08.156757 (+ 2213us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.156830 (+ 73us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.156962 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.179935 1583 consensus_queue.cc:260] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.202924 1583 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:08.203907 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 627) took 1135 ms. Trace:
I20250114 02:54:08.191604 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 624) took 1253 ms. Trace:
I20250114 02:54:08.204634 1736 rpcz_store.cc:276] 0114 02:54:06.937848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.178626 (+1240778us) service_pool.cc:225] Handling call
0114 02:54:08.178978 (+ 352us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.185217 (+ 6239us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.191538 (+ 6321us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.204263 1734 rpcz_store.cc:276] 0114 02:54:07.068650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.203188 (+1134538us) service_pool.cc:225] Handling call
0114 02:54:08.203456 (+ 268us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.203618 (+ 162us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:08.203642 (+ 24us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:08.203681 (+ 39us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:08.203708 (+ 27us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.203757 (+ 49us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.203858 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.213129 1731 raft_consensus.cc:1235] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 02:54:08.214771 2831 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.177592 2593 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.216595 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 622) took 1534 ms. Trace:
I20250114 02:54:08.126118 2732 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.217056 1731 rpcz_store.cc:276] 0114 02:54:06.681829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.176735 (+1494906us) service_pool.cc:225] Handling call
0114 02:54:08.202622 (+ 25887us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:08.216276 (+ 13654us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.216531 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.216301 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 421) took 1702 ms. Trace:
I20250114 02:54:08.188946 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 599) took 1193 ms. Trace:
I20250114 02:54:08.179302 2757 raft_consensus.cc:2758] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250114 02:54:08.219094 1587 rpcz_store.cc:276] 0114 02:54:06.514346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:06.514875 (+ 529us) service_pool.cc:225] Handling call
0114 02:54:06.515263 (+ 388us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 02:54:06.515740 (+ 477us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:06.515778 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 02:54:06.516381 (+ 603us) log.cc:844] Serialized 52 byte log entry
0114 02:54:06.724559 (+208178us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:06.724615 (+ 56us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:06.725936 (+ 1321us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:07.856381 (+1130445us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b440002494c. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce30ad7c5 00007fbce30ae1e2 00007fbce5d63da1 00007fbce5d5f68a 00007fbce5d534ae 00007fbce5d527f6 00007fbce5d52418 00007fbce5d5bdf1 00007fbce5d5bda9 00007fbce5d5bd39
0114 02:54:08.189152 (+332771us) raft_consensus.cc:1680] finished
0114 02:54:08.189179 (+ 27us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.216226 (+ 27047us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":110,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":14709,"spinlock_wait_cycles":46350720,"thread_start_us":106982,"threads_started":1,"wal-append.queue_time_us":275630}
I20250114 02:54:08.228256 1587 raft_consensus.cc:1270] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:08.228755 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 601) took 1209 ms. Trace:
I20250114 02:54:08.229027 1587 rpcz_store.cc:276] 0114 02:54:07.019843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.227680 (+1207837us) service_pool.cc:225] Handling call
0114 02:54:08.227960 (+ 280us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.228553 (+ 593us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.228708 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.165118 1730 rpcz_store.cc:276] 0114 02:54:06.188348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:07.864291 (+1675943us) service_pool.cc:225] Handling call
0114 02:54:07.866148 (+ 1857us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.094672 (+228524us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.094955 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 02:54:08.222544 1701 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.772s user 0.045s sys 0.133s
W20250114 02:54:08.212797 2442 consensus_meta.cc:287] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.874s user 0.008s sys 0.003s
I20250114 02:54:08.230953 2442 raft_consensus.cc:2743] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:08.231355 2732 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:08.231707 2655 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:08.231742 1701 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.832s user 0.046s sys 0.133s
I20250114 02:54:08.232834 2732 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.233354 2732 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.235267 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 618) took 1554 ms. Trace:
I20250114 02:54:08.235054 2647 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.235514 2732 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:08.235667 1728 rpcz_store.cc:276] 0114 02:54:06.680895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.174783 (+1493888us) service_pool.cc:225] Handling call
0114 02:54:08.234709 (+ 59926us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.234908 (+ 199us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:08.234939 (+ 31us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:08.234991 (+ 52us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:08.235027 (+ 36us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.235089 (+ 62us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.235207 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.220892 2512 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:08.237056 2647 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:08.170127 2804 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.162853 2688 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.179247 1581 rpcz_store.cc:276] 0114 02:54:06.994791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.150009 (+1155218us) service_pool.cc:225] Handling call
0114 02:54:08.150389 (+ 380us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.161135 (+ 10746us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:08.166943 (+ 5808us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:08.170617 (+ 3674us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:08.178434 (+ 7817us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.178541 (+ 107us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.178710 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.227329 1579 rpcz_store.cc:276] 0114 02:54:06.995075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.187697 (+1192622us) service_pool.cc:225] Handling call
0114 02:54:08.188003 (+ 306us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.188705 (+ 702us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.188883 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.215299 2831 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.243222 2593 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:08.295835 1733 raft_consensus.cc:1270] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:08.296578 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 631) took 1088 ms. Trace:
I20250114 02:54:08.297133 1654 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:08.297367 2804 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:08.297883 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48636 (request call id 427) took 1093 ms. Trace:
I20250114 02:54:08.298241 1582 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:08.298899 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34928 (request call id 607) took 1090 ms. Trace:
I20250114 02:54:08.246595 1583 raft_consensus.cc:1270] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250114 02:54:08.300206 1583 pending_rounds.cc:77] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 2
I20250114 02:54:08.300320 1587 raft_consensus.cc:1270] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:08.300454 1583 pending_rounds.cc:99] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250114 02:54:08.300704 1583 raft_consensus.cc:2883] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:08.301478 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 597) took 1307 ms. Trace:
I20250114 02:54:08.301844 1583 rpcz_store.cc:276] 0114 02:54:06.994480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.149610 (+1155130us) service_pool.cc:225] Handling call
0114 02:54:08.149876 (+ 266us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.301202 (+151326us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.301417 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.304435 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 431) took 1105 ms. Trace:
I20250114 02:54:08.304883 1583 rpcz_store.cc:276] 0114 02:54:07.199445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.303501 (+1104056us) service_pool.cc:225] Handling call
0114 02:54:08.303831 (+ 330us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.304066 (+ 235us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:08.304103 (+ 37us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:08.304155 (+ 52us) raft_consensus.cc:1640] Marking committed up to 3
0114 02:54:08.304197 (+ 42us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.304261 (+ 64us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.304376 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 02:54:08.299479 1586 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 1.040s user 0.029s sys 0.110s
W20250114 02:54:08.306839 1586 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 1.048s user 0.029s sys 0.110s
W20250114 02:54:08.308502 1702 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.053s user 0.021s sys 0.046s
W20250114 02:54:08.310271 1702 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.055s user 0.022s sys 0.046s
I20250114 02:54:08.310709 1729 raft_consensus.cc:1270] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:08.311439 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 635) took 1054 ms. Trace:
I20250114 02:54:08.311800 1729 rpcz_store.cc:276] 0114 02:54:07.256856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.193884 (+937028us) service_pool.cc:225] Handling call
0114 02:54:08.310167 (+116283us) spinlock_profiling.cc:243] Waited 8.95 ms on lock 0x7b5800023d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:54:08.310356 (+ 189us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.311139 (+ 783us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.311377 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20593536}
I20250114 02:54:08.310786 1583 raft_consensus.cc:1235] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:08.314884 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 434) took 1113 ms. Trace:
I20250114 02:54:08.315429 1583 rpcz_store.cc:276] 0114 02:54:07.201324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.306351 (+1105027us) service_pool.cc:225] Handling call
0114 02:54:08.310636 (+ 4285us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.314479 (+ 3843us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.314813 (+ 334us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.299249 1582 rpcz_store.cc:276] 0114 02:54:07.208973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.117731 (+908758us) service_pool.cc:225] Handling call
0114 02:54:08.298842 (+181111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.316805 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:08.160060 1584 rpcz_store.cc:276] 0114 02:54:06.648353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:06.974040 (+325687us) service_pool.cc:225] Handling call
0114 02:54:07.045187 (+ 71147us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 02:54:07.096947 (+ 51760us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:07.110708 (+ 13761us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 02:54:07.154813 (+ 44105us) log.cc:844] Serialized 52 byte log entry
0114 02:54:07.545949 (+391136us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:07.561987 (+ 16038us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:07.576933 (+ 14946us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:08.105677 (+528744us) raft_consensus.cc:1680] finished
0114 02:54:08.105721 (+ 44us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.105915 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5111,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":24014,"thread_start_us":268503,"threads_started":1,"wal-append.queue_time_us":423227}
I20250114 02:54:08.327975 1584 raft_consensus.cc:1270] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:08.327602 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:08.328685 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 609) took 1110 ms. Trace:
I20250114 02:54:08.329147 1584 rpcz_store.cc:276] 0114 02:54:07.218036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.327272 (+1109236us) service_pool.cc:225] Handling call
0114 02:54:08.327601 (+ 329us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.328387 (+ 786us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.328619 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.317420 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 608) took 1107 ms. Trace:
I20250114 02:54:08.309374 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 588) took 1617 ms. Trace:
I20250114 02:54:08.330637 1586 rpcz_store.cc:276] 0114 02:54:06.691961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:07.100535 (+408574us) service_pool.cc:225] Handling call
0114 02:54:07.100945 (+ 410us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 02:54:07.101358 (+ 413us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:07.101396 (+ 38us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 02:54:07.102009 (+ 613us) log.cc:844] Serialized 52 byte log entry
0114 02:54:07.212474 (+110465us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:08.307283 (+1094809us) log.cc:844] Serialized 14 byte log entry
0114 02:54:08.308972 (+ 1689us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.309046 (+ 74us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:08.309091 (+ 45us) raft_consensus.cc:1680] finished
0114 02:54:08.309108 (+ 17us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.309324 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1678,"raft.queue_time_us":14278,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":10293,"thread_start_us":1157959,"threads_started":2,"wal-append.queue_time_us":319526}
I20250114 02:54:08.333634 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:08.310074 2804 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.296904 1733 rpcz_store.cc:276] 0114 02:54:07.207862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.188859 (+980997us) service_pool.cc:225] Handling call
0114 02:54:08.218667 (+ 29808us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.296291 (+ 77624us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.296519 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.330914 1583 rpcz_store.cc:276] 0114 02:54:07.210118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.316672 (+1106554us) service_pool.cc:225] Handling call
0114 02:54:08.316967 (+ 295us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.317133 (+ 166us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:08.317158 (+ 25us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:08.317205 (+ 47us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:08.317235 (+ 30us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.317285 (+ 50us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.317377 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.298309 1654 rpcz_store.cc:276] 0114 02:54:07.204684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:07.983786 (+779102us) service_pool.cc:225] Handling call
0114 02:54:08.297810 (+314024us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:08.345913 1735 raft_consensus.cc:1270] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250114 02:54:08.347297 2645 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.013s user 0.028s sys 0.044s
W20250114 02:54:08.354270 2645 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.031s user 0.028s sys 0.045s
I20250114 02:54:08.348652 1702 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:08.353394 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 621) took 1671 ms. Trace:
I20250114 02:54:08.355085 1735 rpcz_store.cc:276] 0114 02:54:06.681606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.174535 (+1492929us) service_pool.cc:225] Handling call
0114 02:54:08.345414 (+170879us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b5800023d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:54:08.345598 (+ 184us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.353067 (+ 7469us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.353328 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27999104}
I20250114 02:54:08.364022 1702 leader_election.cc:304] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:08.368923 2837 raft_consensus.cc:2743] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250114 02:54:08.356626 2756 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.012s sys 0.048s
W20250114 02:54:08.311144 2796 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.721s user 0.005s sys 0.020s
I20250114 02:54:08.350122 2593 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:08.351716 1581 raft_consensus.cc:1270] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:08.351975 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 603) took 1331 ms. Trace:
W20250114 02:54:08.374181 2803 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.003s sys 0.012s
W20250114 02:54:08.373517 2756 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.744s user 0.012s sys 0.049s
I20250114 02:54:08.369714 2829 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:08.378178 2796 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.784s user 0.005s sys 0.020s
W20250114 02:54:08.386121 1699 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.899s user 0.026s sys 0.048s
I20250114 02:54:08.380686 1702 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
W20250114 02:54:08.389828 1699 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.094s user 0.027s sys 0.049s
I20250114 02:54:08.352293 2804 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:08.379496 1585 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.003s sys 0.011s
W20250114 02:54:08.412478 1553 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.717s user 0.033s sys 0.087s
W20250114 02:54:08.382622 2643 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.006s sys 0.010s
W20250114 02:54:08.412618 1585 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s user 0.003s sys 0.011s
I20250114 02:54:08.413053 1702 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:08.378932 2839 raft_consensus.cc:2798] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:08.415719 1732 raft_consensus.cc:1270] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:08.416416 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 632) took 1208 ms. Trace:
W20250114 02:54:08.412878 1553 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.718s user 0.033s sys 0.087s
I20250114 02:54:08.416699 1732 rpcz_store.cc:276] 0114 02:54:07.208430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.211858 (+1003428us) service_pool.cc:225] Handling call
0114 02:54:08.415025 (+203167us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b5800023d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:54:08.415271 (+ 246us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.416184 (+ 913us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.416366 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26006784}
W20250114 02:54:08.383683 1580 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.273s user 0.003s sys 0.023s
W20250114 02:54:08.387600 1626 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.790s user 0.009s sys 0.028s
W20250114 02:54:08.114461 1628 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.021s sys 0.050s
W20250114 02:54:08.422833 1628 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.846s user 0.028s sys 0.050s
W20250114 02:54:08.423175 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 855518us
I20250114 02:54:08.423822 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
W20250114 02:54:08.377199 2820 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.000s sys 0.013s
I20250114 02:54:08.426448 1662 raft_consensus.cc:2363] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
I20250114 02:54:08.426842 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:08.427356 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.427958 1662 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:08.394737 2688 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:08.428979 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.433992 1662 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:08.435168 1728 raft_consensus.cc:2388] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:08.435132 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:08.413972 2643 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s user 0.006s sys 0.010s
I20250114 02:54:08.438176 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.438480 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.476135 1658 raft_consensus.cc:2463] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
W20250114 02:54:08.478323 1660 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.033s sys 0.068s
W20250114 02:54:08.430220 1626 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.840s user 0.009s sys 0.028s
W20250114 02:54:08.415989 1700 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.055s user 0.032s sys 0.057s
W20250114 02:54:08.381129 2803 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.526s user 0.003s sys 0.013s
I20250114 02:54:08.418849 2839 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:08.445665 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:08.483669 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.478292 1628 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
W20250114 02:54:08.422394 1580 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.312s user 0.003s sys 0.023s
I20250114 02:54:08.380280 2593 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:08.483640 1660 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s user 0.033s sys 0.068s
I20250114 02:54:08.491796 1662 raft_consensus.cc:2388] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 5: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
W20250114 02:54:08.378392 2513 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.007s sys 0.007s
I20250114 02:54:08.445277 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:08.430384 2836 raft_consensus.cc:2798] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:08.487797 1586 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:08.432359 2807 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
W20250114 02:54:08.452665 2743 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.000s sys 0.013s
I20250114 02:54:08.380403 1579 rpcz_store.cc:276] 0114 02:54:07.020425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.298095 (+1277670us) service_pool.cc:225] Handling call
0114 02:54:08.298460 (+ 365us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.308756 (+ 10296us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:08.309855 (+ 1099us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:08.315042 (+ 5187us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:08.316318 (+ 1276us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.339707 (+ 23389us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.350282 (+ 10575us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 02:54:08.431520 2820 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.652s user 0.000s sys 0.014s
I20250114 02:54:08.394987 2829 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.511319 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.482812 2732 raft_consensus.cc:2798] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:08.600526 2645 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.511943 1628 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:08.512605 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:08.489993 1700 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.131s user 0.032s sys 0.061s
I20250114 02:54:08.511654 2836 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 02:54:08.580104 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 90476.3us
I20250114 02:54:08.619580 1553 leader_election.cc:400] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
I20250114 02:54:08.619889 1553 leader_election.cc:403] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250114 02:54:08.620671 1553 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:08.581301 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 427) took 1941 ms. Trace:
I20250114 02:54:08.580367 2593 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:08.517107 2197 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.511463 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.586817 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 604) took 1538 ms. Trace:
W20250114 02:54:08.589628 2834 log.cc:926] Time spent T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.241s user 0.000s sys 0.001s
I20250114 02:54:08.593078 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 602) took 1573 ms. Trace:
I20250114 02:54:08.594973 2508 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:08.595484 2796 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.595973 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.597028 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:08.485481 2831 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:08.500610 2513 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.713s user 0.007s sys 0.007s
I20250114 02:54:08.492246 2840 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.478415 2728 consensus_queue.cc:1035] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.085s
I20250114 02:54:08.494845 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:08.490576 2839 raft_consensus.cc:3054] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:08.511205 2655 raft_consensus.cc:2798] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:08.428474 2835 consensus_queue.cc:1046] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.083s
I20250114 02:54:08.630395 2593 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.630234 2442 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.570111 1653 raft_consensus.cc:1235] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:08.610134 2807 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.605057 1702 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:08.513549 1583 raft_consensus.cc:2405] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:08.603056 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:08.817574 1586 raft_consensus.cc:2463] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:08.818414 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:54:08.616264 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 104424us
I20250114 02:54:08.610939 2811 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:08.604525 2743 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.881s user 0.003s sys 0.014s
I20250114 02:54:08.617278 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:08.618295 1659 raft_consensus.cc:1235] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
I20250114 02:54:08.623052 2197 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.626405 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.827160 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:08.828059 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.828631 1662 raft_consensus.cc:2463] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:08.829428 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.622550 1553 leader_election.cc:400] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
I20250114 02:54:08.626328 1587 rpcz_store.cc:276] 0114 02:54:07.020115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.229639 (+1209524us) service_pool.cc:225] Handling call
0114 02:54:08.229876 (+ 237us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.300645 (+ 70769us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.592795 (+292150us) spinlock_profiling.cc:243] Waited 70.1 ms on lock 0x7b54001ab3d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:08.593010 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":161234816}
I20250114 02:54:08.914134 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:08.574237 2512 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:08.622745 1585 rpcz_store.cc:276] 0114 02:54:06.640223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:06.680459 (+ 40236us) service_pool.cc:225] Handling call
0114 02:54:06.680740 (+ 281us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:06.681684 (+ 944us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:06.681710 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:06.682093 (+ 383us) log.cc:844] Serialized 26 byte log entry
0114 02:54:07.869091 (+1186998us) spinlock_profiling.cc:243] Waited 5.91 ms on lock 0x7b1800227b60. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce317e565 00007fbcebd907f9 00007fbce319ba6b 00007fbce3199e6f 00007fbce319bfd2 00007fbce5d51e4b 00007fbce5d57913 00007fbce5d57cc1 00007fbce60a945f
0114 02:54:08.414979 (+545888us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:08.415026 (+ 47us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.415110 (+ 84us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:08.580965 (+165855us) raft_consensus.cc:1680] finished
0114 02:54:08.580991 (+ 26us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.581234 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13193,"raft.queue_time_us":665,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":4058,"spinlock_wait_cycles":13598464,"thread_start_us":545704,"threads_started":1,"wal-append.queue_time_us":1795452}
I20250114 02:54:08.925376 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:08.631124 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:08.603828 2824 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.882s user 0.010s sys 0.033s
I20250114 02:54:08.829921 2811 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.626896 2643 raft_consensus.cc:2798] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:08.490101 2647 consensus_queue.cc:1035] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.108s
I20250114 02:54:08.630008 2837 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:08.626701 2836 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:08.661177 2840 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.703169 2710 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.698467 1582 raft_consensus.cc:2405] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 4 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250114 02:54:08.723357 2835 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:08.666478 1730 raft_consensus.cc:1235] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:08.673271 2757 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:08.717555 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.719890 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:08.718920 2846 log.cc:926] Time spent T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20250114 02:54:08.682827 2442 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.721038 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.740720 2804 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:08.663959 2655 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:08.672844 2593 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.715332 2806 consensus_queue.cc:1035] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:08.632566 2508 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.717119 2844 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:08.664875 2831 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:08.716428 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 589) took 2021 ms. Trace:
I20250114 02:54:08.653651 2796 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.715957 2799 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.627104 2645 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.652864 2732 raft_consensus.cc:695] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:08.653614 2693 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:08.626109 1581 rpcz_store.cc:276] 0114 02:54:07.048618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.298789 (+1250171us) service_pool.cc:225] Handling call
0114 02:54:08.299143 (+ 354us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.379424 (+ 80281us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.586479 (+207055us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b540016dd54. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:08.586745 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35988224}
I20250114 02:54:08.748734 2829 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:08.816879 1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 492) took 1673 ms. Trace:
I20250114 02:54:08.856735 1482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39732 (request call id 92) took 5487 ms. Trace:
I20250114 02:54:08.813633 2838 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:08.880523 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:08.850032 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:08.830587 2832 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:08.851117 2197 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:08.915128 1553 leader_election.cc:403] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20250114 02:54:08.628058 2528 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.019s sys 0.025s
W20250114 02:54:08.930054 2824 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.212s user 0.010s sys 0.034s
W20250114 02:54:09.020349 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 397950us
W20250114 02:54:09.018149 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 191255us
I20250114 02:54:09.021145 1553 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:08.932528 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48636 (request call id 414) took 2647 ms. Trace:
I20250114 02:54:09.007884 1661 rpcz_store.cc:276] 0114 02:54:07.143187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.008343 (+865156us) service_pool.cc:225] Handling call
0114 02:54:08.008615 (+ 272us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.008819 (+ 204us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 02:54:08.473764 (+464945us) log.cc:844] Serialized 14 byte log entry
0114 02:54:08.816581 (+342817us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:08.816626 (+ 45us) raft_consensus.cc:1640] Marking committed up to 3
0114 02:54:08.816661 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.816719 (+ 58us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.816831 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":596450,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":9973,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":24957,"thread_start_us":727004,"threads_started":2,"wal-append.queue_time_us":464399}
I20250114 02:54:08.935695 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.897958 2843 raft_consensus.cc:2798] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:08.886265 2688 consensus_queue.cc:1035] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20250114 02:54:08.998445 2655 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:08.976353 1586 raft_consensus.cc:2388] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:08.965389 2643 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:08.976119 2807 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:09.026997 2528 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.125s user 0.020s sys 0.025s
I20250114 02:54:08.937790 2512 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.982739 1578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 600) took 1963 ms. Trace:
I20250114 02:54:09.020628 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:08.930629 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.022105 1553 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:09.027449 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:09.034448 2803 raft_consensus.cc:2798] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:09.025162 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.009341 1482 rpcz_store.cc:276] 0114 02:54:03.369124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:03.369331 (+ 207us) service_pool.cc:225] Handling call
0114 02:54:08.856661 (+5487330us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:06.122842 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:06.123412 (+ 570us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:06.123443 (+ 31us) write_op.cc:435] Acquired schema lock
0114 02:54:06.123464 (+ 21us) tablet.cc:659] Decoding operations
0114 02:54:06.269319 (+145855us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:06.269394 (+ 75us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:06.269424 (+ 30us) tablet.cc:682] Acquiring locks for 21 operations
0114 02:54:06.270887 (+ 1463us) tablet.cc:698] Row locks acquired
0114 02:54:06.270906 (+ 19us) write_op.cc:260] PREPARE: finished
0114 02:54:06.271031 (+ 125us) write_op.cc:270] Start()
0114 02:54:06.271138 (+ 107us) write_op.cc:276] Timestamp: P: 1736823246270993 usec, L: 0
0114 02:54:06.271154 (+ 16us) op_driver.cc:348] REPLICATION: starting
0114 02:54:06.340906 (+ 69752us) log.cc:844] Serialized 8238 byte log entry
0114 02:54:06.977875 (+636969us) op_driver.cc:464] REPLICATION: finished
0114 02:54:07.354117 (+376242us) write_op.cc:301] APPLY: starting
0114 02:54:07.361090 (+ 6973us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:07.617974 (+256884us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:07.628348 (+ 10374us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:07.732136 (+103788us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:07.732530 (+ 394us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=21
0114 02:54:07.732594 (+ 64us) write_op.cc:312] APPLY: finished
0114 02:54:07.894176 (+161582us) log.cc:844] Serialized 144 byte log entry
0114 02:54:08.412479 (+518303us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:08.431391 (+ 18912us) write_op.cc:454] Released schema lock
0114 02:54:08.469596 (+ 38205us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:08.579137 (+109541us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:08.579169 (+ 32us) write_op.cc:454] Released schema lock
0114 02:54:08.618360 (+ 39191us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b440002494c. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce30ad7c5 00007fbce30ae1e2 00007fbce5d63da1 00007fbce5d5f68a 00007fbce5d534ae 00007fbce5d527f6 00007fbce5d52418 00007fbce5d5bdf1 00007fbce5d5bda9 00007fbce5d5bd39
Metrics: {"reported_tablets":54,"child_traces":[["op",{"apply.queue_time_us":286981,"apply.run_cpu_time_us":25405,"apply.run_wall_time_us":1241980,"num_ops":21,"prepare.queue_time_us":378319,"prepare.run_cpu_time_us":53736,"prepare.run_wall_time_us":335261,"raft.queue_time_us":213181,"raft.run_cpu_time_us":7844,"raft.run_wall_time_us":282849,"replication_time_us":664752,"spinlock_wait_cycles":5382912,"thread_start_us":1207167,"threads_started":5,"wal-append.queue_time_us":756698,"wal-append.run_cpu_time_us":9813,"wal-append.run_wall_time_us":1235375}]]}
I20250114 02:54:09.028046 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.022782 1660 rpcz_store.cc:276] 0114 02:54:06.285453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:06.285598 (+ 145us) service_pool.cc:225] Handling call
0114 02:54:06.290370 (+ 4772us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:06.292063 (+ 1693us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:06.292090 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:06.292445 (+ 355us) log.cc:844] Serialized 26 byte log entry
0114 02:54:07.895680 (+1603235us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b180006e2c0. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce317e565 00007fbcebd907f9 00007fbce319ba6b 00007fbce3199e6f 00007fbce319bfd2 00007fbce5d51e4b 00007fbce5d57913 00007fbce5d57cc1 00007fbce60a945f
0114 02:54:08.493927 (+598247us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:08.493989 (+ 62us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.494068 (+ 79us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:08.932131 (+438063us) raft_consensus.cc:1680] finished
0114 02:54:08.932173 (+ 42us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.932450 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22184,"raft.queue_time_us":1497,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":2092,"spinlock_wait_cycles":3143424,"thread_start_us":598004,"threads_started":1,"wal-append.queue_time_us":2562708}
I20250114 02:54:08.966603 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:08.977531 2710 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.975327 2837 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:09.044059 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:54:09.032464 2829 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:09.032130 2804 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
W20250114 02:54:08.997397 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 397118us
I20250114 02:54:09.019894 2693 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:08.981907 2835 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.053483 1553 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:09.024930 2197 consensus_queue.cc:1035] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:09.065200 1583 raft_consensus.cc:2463] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:08.996989 2757 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.009093 2831 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.009547 2799 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.051678 1662 raft_consensus.cc:1270] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:54:09.040884 2643 raft_consensus.cc:3054] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.044548 2839 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.096698 1628 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:09.092622 1660 raft_consensus.cc:1270] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:54:09.092958 2197 consensus_queue.cc:1035] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:09.099098 2197 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:09.099370 2197 raft_consensus.cc:695] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:09.057080 2528 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:09.056383 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.064057 2824 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:09.066044 1582 raft_consensus.cc:3054] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.102231 1587 raft_consensus.cc:2405] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:09.099829 2197 consensus_queue.cc:237] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.063457 2803 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:09.106771 1585 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:09.068898 2870 raft_consensus.cc:2798] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:09.075049 2829 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.113404 2831 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.087044 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.114348 2829 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.114730 1581 raft_consensus.cc:2463] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:09.109695 2528 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.115319 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.115892 1660 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:09.116670 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.116845 2528 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.117221 1660 raft_consensus.cc:2463] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:09.108831 2870 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:09.032711 2838 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.102553 2730 consensus_queue.cc:1046] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20250114 02:54:09.135264 2730 raft_consensus.cc:3049] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:09.135919 2730 raft_consensus.cc:738] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:09.090963 2385 raft_consensus.cc:2798] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:09.074935 2647 consensus_queue.cc:1035] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20250114 02:54:09.075476 2756 consensus_queue.cc:1035] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20250114 02:54:09.003100 2844 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.150966 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.141494 2385 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:09.166961 2442 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.125085 2743 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:09.075289 2732 consensus_queue.cc:237] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.111326 2803 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:09.110368 2824 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:09.103616 2864 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:09.171648 2857 log.cc:926] Time spent T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.203s user 0.001s sys 0.000s
I20250114 02:54:09.102983 2845 consensus_queue.cc:1046] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.077s
I20250114 02:54:09.186303 1658 raft_consensus.cc:1235] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4. Ops: []
W20250114 02:54:09.139489 2854 log.cc:926] Time spent T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.226s user 0.000s sys 0.001s
I20250114 02:54:09.178128 2385 raft_consensus.cc:3054] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:09.170893 2811 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:09.217626 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 146419us
W20250114 02:54:09.138124 2861 log.cc:926] Time spent T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.158s user 0.000s sys 0.001s
I20250114 02:54:09.137598 2730 consensus_queue.cc:260] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.124876 2870 raft_consensus.cc:3054] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
W20250114 02:54:09.211992 2858 log.cc:926] Time spent T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.218s user 0.000s sys 0.001s
I20250114 02:54:09.175120 2508 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.188180 2840 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:09.231104 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 133507us
I20250114 02:54:09.142993 2688 consensus_queue.cc:1035] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.088s
I20250114 02:54:09.241322 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:09.242127 1731 raft_consensus.cc:3054] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:09.243039 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:09.243472 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:09.243819 1729 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:09.244095 1728 raft_consensus.cc:3054] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:09.186177 2824 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.219756 2593 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.250375 1728 raft_consensus.cc:2463] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:09.252741 2824 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.266057 1731 raft_consensus.cc:2463] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:09.266396 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:09.267269 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.268463 2807 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:09.268803 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.284423 1736 raft_consensus.cc:2405] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 02:54:09.269994 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.284978 1733 raft_consensus.cc:2463] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:09.285753 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.223630 2873 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:09.211158 2743 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.257877 1729 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:09.232548 2645 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.280462 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:09.292448 1731 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.279552 1734 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:09.286284 1733 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:09.237497 2796 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.298485 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:09.300659 1728 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
W20250114 02:54:09.298653 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 77464.5us
I20250114 02:54:09.302807 1702 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:09.303884 1702 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:09.282910 2878 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:09.271559 2512 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.271934 2728 consensus_queue.cc:1035] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20250114 02:54:09.308877 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.309355 1702 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:09.318079 1731 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:09.321192 1702 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:09.292583 2693 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:09.326474 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 98557.4us
I20250114 02:54:09.231225 2871 consensus_queue.cc:1046] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.064s
I20250114 02:54:09.001022 1580 rpcz_store.cc:276] 0114 02:54:06.694745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:07.108511 (+413766us) service_pool.cc:225] Handling call
0114 02:54:07.108893 (+ 382us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:07.109288 (+ 395us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:07.109340 (+ 52us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:07.109877 (+ 537us) log.cc:844] Serialized 26 byte log entry
0114 02:54:08.558949 (+1449072us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:08.578066 (+ 19117us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.578209 (+ 143us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:08.716217 (+138008us) raft_consensus.cc:1680] finished
0114 02:54:08.716236 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.716373 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":216,"raft.queue_time_us":20892,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":7346,"thread_start_us":1383077,"threads_started":1,"wal-append.queue_time_us":1522437}
I20250114 02:54:09.218685 2864 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:09.307425 2859 log.cc:926] Time spent T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.237s user 0.000s sys 0.001s
I20250114 02:54:09.295732 2710 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.294036 2840 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:09.374236 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:09.370683 2865 log.cc:926] Time spent T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.205s user 0.001s sys 0.000s
I20250114 02:54:09.367933 2513 raft_consensus.cc:3049] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:09.298116 2870 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.326939 2512 raft_consensus.cc:2798] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:09.295934 2873 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:09.367280 2876 log.cc:926] Time spent T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20250114 02:54:09.358057 2754 consensus_queue.cc:1046] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 02:54:09.354219 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 446) took 1010 ms. Trace:
I20250114 02:54:09.469497 2873 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:09.336133 2866 log.cc:926] Time spent T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.241s user 0.000s sys 0.001s
I20250114 02:54:09.469870 1730 rpcz_store.cc:276] 0114 02:54:08.344266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.345387 (+ 1121us) service_pool.cc:225] Handling call
0114 02:54:08.372508 (+ 27121us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.981628 (+609120us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:09.341942 (+360314us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b54003d3bd4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:09.354154 (+ 12212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":593309952}
I20250114 02:54:09.323992 2796 raft_consensus.cc:2798] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:09.471472 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.472455 1730 raft_consensus.cc:2388] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:09.473439 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.363296 2844 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:09.326869 2862 log.cc:926] Time spent T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.220s user 0.000s sys 0.001s
I20250114 02:54:09.374004 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:09.365590 2843 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:09.367738 2811 raft_consensus.cc:3049] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:54:09.479163 2843 raft_consensus.cc:3054] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.479297 2811 raft_consensus.cc:738] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:09.480149 2811 consensus_queue.cc:260] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.481273 2811 raft_consensus.cc:3054] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:09.348078 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.483498 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.484047 1736 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:09.484768 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.328780 2839 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.326243 2757 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.385654 1662 raft_consensus.cc:2463] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:09.050467 1578 rpcz_store.cc:276] 0114 02:54:07.019506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.188027 (+1168521us) service_pool.cc:225] Handling call
0114 02:54:08.188361 (+ 334us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:08.188554 (+ 193us) raft_consensus.cc:1529] Early marking committed up to index 2
0114 02:54:08.410383 (+221829us) log.cc:844] Serialized 14 byte log entry
0114 02:54:08.924780 (+514397us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:08.927551 (+ 2771us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:08.947372 (+ 19821us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.968427 (+ 21055us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:08.980248 (+ 11821us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43268,"prepare.queue_time_us":283781,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":40,"raft.queue_time_us":4466,"spinlock_wait_cycles":107696512,"thread_start_us":675755,"threads_started":2,"wal-append.queue_time_us":624210}
I20250114 02:54:09.474823 2844 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:09.399873 2838 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:09.400357 2847 log.cc:926] Time spent T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.534s user 0.003s sys 0.000s
I20250114 02:54:09.487013 2844 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.400790 2824 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.339327 2508 raft_consensus.cc:2798] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:09.488246 2508 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:09.488642 2508 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:09.489161 2844 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.328637 2799 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.402875 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.403196 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.491664 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:09.492326 1735 raft_consensus.cc:2363] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
I20250114 02:54:09.492961 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.494227 1735 raft_consensus.cc:1235] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:09.404179 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.494956 1734 raft_consensus.cc:2388] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:09.496052 1734 raft_consensus.cc:1270] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:09.420814 1731 raft_consensus.cc:1270] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:54:09.307262 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:09.339830 2835 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.459506 1727 raft_consensus.cc:1235] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:09.342485 2837 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:09.461704 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 132823us
I20250114 02:54:09.503245 2887 raft_consensus.cc:3049] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:09.503626 2887 raft_consensus.cc:738] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:09.504432 2887 consensus_queue.cc:260] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.505499 1583 raft_consensus.cc:1235] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:09.505621 2887 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.506151 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 440) took 1061 ms. Trace:
I20250114 02:54:09.506490 1583 rpcz_store.cc:276] 0114 02:54:08.444460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:09.079603 (+635143us) service_pool.cc:225] Handling call
0114 02:54:09.505188 (+425585us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:54:09.505356 (+ 168us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:09.505941 (+ 585us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:09.506102 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23839616}
I20250114 02:54:09.507602 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:09.509960 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 496) took 1090 ms. Trace:
I20250114 02:54:09.511612 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 443) took 1053 ms. Trace:
I20250114 02:54:09.511978 1587 rpcz_store.cc:276] 0114 02:54:08.458332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.913899 (+455567us) service_pool.cc:225] Handling call
0114 02:54:09.511440 (+597541us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b540037aad4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:09.511558 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":430696832}
I20250114 02:54:09.513062 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.514111 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:09.496461 2655 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.526568 2833 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:09.526684 1553 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:09.527320 2655 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.527509 2881 raft_consensus.cc:2743] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:09.527267 2833 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.528626 1553 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:09.529440 2870 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.530056 2833 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.530258 2891 raft_consensus.cc:2743] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:09.531229 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.532657 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:09.533202 2655 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:09.533792 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.401794 2840 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.534724 2835 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:09.535251 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.535621 1553 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:09.535537 2806 consensus_queue.cc:1035] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:09.536314 2840 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.536479 1553 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:09.536902 2806 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:09.537236 2881 raft_consensus.cc:2798] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:09.537809 2881 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:09.538120 2881 raft_consensus.cc:3054] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.467293 2743 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.467425 2832 raft_consensus.cc:2743] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
I20250114 02:54:09.538641 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.468727 2864 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.540532 2832 consensus_queue.cc:1046] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:09.471560 2796 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:09.541343 2824 raft_consensus.cc:3049] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:09.541429 2796 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.541759 2824 raft_consensus.cc:738] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:09.541986 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.460481 2512 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:09.542641 2512 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.542536 2824 consensus_queue.cc:260] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.543504 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.543699 2824 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.544080 1734 raft_consensus.cc:2463] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:09.545871 1734 raft_consensus.cc:1235] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250114 02:54:09.308393 2645 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:09.546716 2645 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:09.547060 2645 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
W20250114 02:54:09.286957 1628 connection.cc:232] RPC call timeout handler was delayed by -1.7568257039999935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1713s for a response
I20250114 02:54:09.461966 1728 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:09.401496 2513 raft_consensus.cc:738] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:09.289644 2807 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.590044 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.590639 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 445) took 1131 ms. Trace:
I20250114 02:54:09.590983 1585 rpcz_store.cc:276] 0114 02:54:08.458931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.925073 (+466142us) service_pool.cc:225] Handling call
0114 02:54:09.590468 (+665395us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b540012dc54. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:09.590584 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":415612288}
I20250114 02:54:09.591286 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.591640 2807 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.591921 1656 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:09.592731 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.593886 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:09.540861 2836 consensus_meta.cc:287] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.562s user 0.007s sys 0.001s
I20250114 02:54:09.541080 1732 raft_consensus.cc:1235] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:09.551728 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.601366 1728 raft_consensus.cc:2463] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:09.551920 1662 raft_consensus.cc:1235] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:09.535251 2835 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.537276 2806 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:09.320523 2878 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.608301 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.608942 1731 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:09.615422 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.575706 1587 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 5
I20250114 02:54:09.584076 1582 raft_consensus.cc:2463] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:09.486544 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:09.588354 2730 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:09.588435 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.643898 2891 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:09.644209 1734 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:09.588683 1628 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:09.644506 2891 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:09.645380 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 56766us
I20250114 02:54:09.589519 2513 consensus_queue.cc:260] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.595762 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 446) took 1120 ms. Trace:
I20250114 02:54:09.648526 1586 rpcz_store.cc:276] 0114 02:54:08.475842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.818154 (+342312us) service_pool.cc:225] Handling call
0114 02:54:09.595596 (+777442us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b5400175554. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:09.595710 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":361525120}
I20250114 02:54:09.591943 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.533677 2655 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.594888 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.601774 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.601114 1553 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:09.603230 2830 raft_consensus.cc:3049] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:09.526520 1655 raft_consensus.cc:2388] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:09.601388 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.621212 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.652405 1736 raft_consensus.cc:2463] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:09.621876 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
W20250114 02:54:09.651868 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 50843.6us
I20250114 02:54:09.605273 2836 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.624125 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.510372 1654 rpcz_store.cc:276] 0114 02:54:08.419477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.426812 (+ 7335us) service_pool.cc:225] Handling call
0114 02:54:08.716009 (+289197us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x7b5800002d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315ed0e 0000000000353db9 00007fbceafbea50 00007fbceb084814 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:54:08.716168 (+ 159us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:09.132022 (+415854us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:09.139183 (+ 7161us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:09.164870 (+ 25687us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:09.168166 (+ 3296us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:09.509707 (+341541us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b540021f2d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:09.509762 (+ 55us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:09.509903 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":916151040}
I20250114 02:54:09.615233 2806 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:09.594491 2843 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:09.628942 2804 consensus_meta.cc:287] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.554s user 0.001s sys 0.007s
I20250114 02:54:09.663610 2804 raft_consensus.cc:2743] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
I20250114 02:54:09.663719 1553 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:09.632845 2645 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.664836 2843 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.665055 2790 raft_consensus.cc:2743] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:09.665465 2833 consensus_queue.cc:1035] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:09.656365 2754 raft_consensus.cc:2798] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:09.666977 2754 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:09.667354 2754 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.667594 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.667989 2655 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.668443 1553 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:09.669173 2593 consensus_queue.cc:1046] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:09.670089 2593 raft_consensus.cc:2743] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:09.670630 2593 raft_consensus.cc:3049] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:09.670949 2593 raft_consensus.cc:738] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:09.671280 2891 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.671661 2593 consensus_queue.cc:260] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.672123 2806 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.672986 2593 raft_consensus.cc:3054] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:09.652029 2830 raft_consensus.cc:738] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:09.674664 1553 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:09.634838 2835 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.599819 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:09.618956 2875 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:09.650875 1656 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:09.653542 1702 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:09.654798 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.620635 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 447) took 1140 ms. Trace:
I20250114 02:54:09.687058 1582 rpcz_store.cc:276] 0114 02:54:08.480370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.996870 (+516500us) service_pool.cc:225] Handling call
0114 02:54:09.620572 (+623702us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:09.687215 1702 leader_election.cc:304] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:09.688037 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.657629 1587 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:09.688645 1582 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:09.688879 1702 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:09.657982 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:09.658517 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.674754 1580 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:09.674346 2830 consensus_queue.cc:260] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.674937 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:09.675339 2513 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:09.675216 2884 consensus_queue.cc:1035] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20250114 02:54:09.676339 2757 consensus_queue.cc:1046] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:09.677520 2841 consensus_queue.cc:1035] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:09.700402 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:09.679351 2756 consensus_queue.cc:1035] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20250114 02:54:09.680457 2860 log.cc:926] Time spent T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.658s user 0.000s sys 0.001s
I20250114 02:54:09.683576 2806 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.636655 2796 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.666280 2645 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.689486 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.689599 1658 raft_consensus.cc:3054] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.690061 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.722693 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.695144 2732 raft_consensus.cc:2743] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:09.684640 2838 consensus_queue.cc:1035] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20250114 02:54:09.690588 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.727480 1586 raft_consensus.cc:2405] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:09.705746 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.728304 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.728827 1586 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:09.729640 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.634155 2803 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.690883 1702 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:09.735489 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:09.736461 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 45646.2us
I20250114 02:54:09.737294 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.737501 1702 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:09.655112 2867 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.701892 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.684967 2888 raft_consensus.cc:2798] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:09.682087 1553 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:09.689960 2875 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.692971 2655 consensus_queue.cc:1035] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20250114 02:54:09.695142 2837 raft_consensus.cc:2798] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:09.706241 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34938 (request call id 438) took 1262 ms. Trace:
I20250114 02:54:09.690270 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:09.692759 2442 raft_consensus.cc:2743] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:09.690057 1733 raft_consensus.cc:2388] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:09.704123 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.729350 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.705317 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48632 (request call id 499) took 1284 ms. Trace:
I20250114 02:54:09.722896 2885 consensus_queue.cc:1035] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.012s
I20250114 02:54:09.736547 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.737752 1658 raft_consensus.cc:2463] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:09.697654 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.698372 1583 raft_consensus.cc:2463] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
W20250114 02:54:09.714232 2643 consensus_meta.cc:287] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.603s user 0.004s sys 0.006s
I20250114 02:54:09.802428 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.739432 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:09.740633 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 58664us
I20250114 02:54:09.740625 1702 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:09.803757 1553 leader_election.cc:400] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
W20250114 02:54:09.804095 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 64501.1us
I20250114 02:54:09.804541 1553 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:09.804769 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.705771 2874 consensus_queue.cc:1046] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.089s
I20250114 02:54:09.742501 2888 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:09.781564 2875 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.802953 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:09.806264 1660 raft_consensus.cc:2388] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 5: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:09.806434 1583 raft_consensus.cc:3054] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:09.743818 2806 raft_consensus.cc:2743] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:09.741971 1628 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
W20250114 02:54:09.730494 2385 consensus_meta.cc:287] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.500s user 0.000s sys 0.010s
I20250114 02:54:09.808694 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.808985 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.740886 1587 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:09.702742 2756 consensus_queue.cc:1225] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer b89550620e1d455d887be72232bb5454 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 02:54:09.704494 2830 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:09.706507 2878 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.707078 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.731671 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.791493 2803 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.792573 2508 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.703526 2728 consensus_queue.cc:1035] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20250114 02:54:09.782835 1585 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:09.710227 2757 raft_consensus.cc:3049] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:09.799911 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.738348 2732 raft_consensus.cc:2743] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:09.796418 1584 rpcz_store.cc:276] 0114 02:54:08.443883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.445788 (+ 1905us) service_pool.cc:225] Handling call
0114 02:54:08.489577 (+ 43789us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:08.576834 (+ 87257us) raft_consensus.cc:1529] Early marking committed up to index 1
0114 02:54:08.576878 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:08.577371 (+ 493us) log.cc:844] Serialized 26 byte log entry
0114 02:54:08.880408 (+303037us) raft_consensus.cc:1640] Marking committed up to 1
0114 02:54:08.880451 (+ 43us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:08.880543 (+ 92us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:09.679925 (+799382us) spinlock_profiling.cc:243] Waited 6.81 ms on lock 0x7b440002494c. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce30ad7c5 00007fbce30ae1e2 00007fbce5d63da1 00007fbce5d5f68a 00007fbce5d534ae 00007fbce5d527f6 00007fbce5d52418 00007fbce5d5bdf1 00007fbce5d5bda9 00007fbce5d5bd39
0114 02:54:09.705976 (+ 26051us) raft_consensus.cc:1680] finished
0114 02:54:09.705998 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:09.706187 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":86815,"raft.queue_time_us":87932,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":116,"spinlock_wait_cycles":15660288,"thread_start_us":302751,"threads_started":1,"wal-append.queue_time_us":412692}
I20250114 02:54:09.802932 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.819870 1654 raft_consensus.cc:2388] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:09.805478 2643 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.711927 2836 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625)
I20250114 02:54:09.782769 1579 raft_consensus.cc:2463] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:09.835315 2688 raft_consensus.cc:3049] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:09.835736 2688 raft_consensus.cc:738] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:09.836179 2643 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.836573 2688 consensus_queue.cc:260] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.837718 2688 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:09.821213 2874 raft_consensus.cc:2798] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:09.839083 2874 raft_consensus.cc:695] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:09.839951 2874 consensus_queue.cc:237] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.811976 1702 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:09.810370 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.812594 1662 raft_consensus.cc:1235] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:09.814087 1553 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:09.810969 1653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48636 (request call id 444) took 1095 ms. Trace:
I20250114 02:54:09.739841 2841 consensus_queue.cc:1035] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20250114 02:54:09.815243 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.813918 1583 raft_consensus.cc:2463] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:09.729588 2881 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.742662 2884 consensus_queue.cc:1035] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20250114 02:54:09.864307 2811 raft_consensus.cc:2743] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:09.869125 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:09.816220 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 74300us
I20250114 02:54:09.820959 2871 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:09.816745 1660 raft_consensus.cc:1235] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250114 02:54:09.743793 2838 consensus_queue.cc:1035] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20250114 02:54:09.871088 2871 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.861596 1653 rpcz_store.cc:276] 0114 02:54:08.715982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:09.038693 (+322711us) service_pool.cc:225] Handling call
0114 02:54:09.699606 (+660913us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b5800002d98. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0341a0 00007fbceb033a84 00007fbce4701c14 00007fbce4701ba2
0114 02:54:09.810300 (+110694us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8169344}
I20250114 02:54:09.815778 1582 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:09.824725 2754 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.873203 2871 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.873751 2754 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.797356 1655 rpcz_store.cc:276] 0114 02:54:08.421330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:08.427815 (+ 6485us) service_pool.cc:225] Handling call
0114 02:54:09.704056 (+1276241us) spinlock_profiling.cc:243] Waited 911 ms on lock 0x7b54002b8c54. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:09.705270 (+ 1214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2094889856}
I20250114 02:54:09.819321 2593 consensus_queue.cc:1035] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.013s
I20250114 02:54:09.822980 2508 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.827630 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.875867 1585 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:09.877470 1582 raft_consensus.cc:1270] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:09.877956 1585 raft_consensus.cc:1270] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250114 02:54:09.878335 1585 pending_rounds.cc:77] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 1
I20250114 02:54:09.819121 2710 consensus_queue.cc:1035] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20250114 02:54:09.823956 1580 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:09.881637 1580 raft_consensus.cc:1235] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:09.780381 2512 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.871860 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:09.743183 2645 raft_consensus.cc:2743] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:09.806982 2888 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:09.890923 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:09.891784 1657 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:09.892424 1628 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:09.810195 2442 raft_consensus.cc:2743] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:09.831298 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.894683 2838 raft_consensus.cc:2798] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:09.895010 1586 raft_consensus.cc:2463] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:09.895171 2838 raft_consensus.cc:695] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:09.895996 2838 consensus_queue.cc:237] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.837122 1661 raft_consensus.cc:2388] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:09.897666 2888 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.899274 1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48636 (request call id 429) took 1458 ms. Trace:
I20250114 02:54:09.899631 1661 rpcz_store.cc:276] 0114 02:54:08.440530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:09.024857 (+584327us) service_pool.cc:225] Handling call
0114 02:54:09.899100 (+874243us) spinlock_profiling.cc:243] Waited 764 ms on lock 0x7b54000438d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:09.899219 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1756451584}
I20250114 02:54:09.899991 2888 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:09.797576 2837 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:09.851224 1702 leader_election.cc:304] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:09.851521 2831 raft_consensus.cc:2743] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:09.853340 1578 raft_consensus.cc:2463] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:09.832744 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:09.820168 2513 consensus_queue.cc:1035] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20250114 02:54:09.815583 2385 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.854908 2796 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.855720 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:09.856431 2836 raft_consensus.cc:2798] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:09.863147 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.820784 2757 raft_consensus.cc:738] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:09.863168 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.864981 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:09.920198 1584 raft_consensus.cc:2388] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 5: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:09.919934 1482 catalog_manager.cc:5526] T d43434bae9eb4b2fb3699343149bb385 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 3 to 4, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 4 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:09.865744 1553 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:09.883541 1585 pending_rounds.cc:99] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250114 02:54:09.921689 1585 raft_consensus.cc:2883] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:09.888099 2512 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.889053 1580 raft_consensus.cc:1270] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 02:54:09.902407 1628 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
W20250114 02:54:09.906597 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 55430.9us
I20250114 02:54:09.913431 2442 raft_consensus.cc:2798] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:09.926353 2442 raft_consensus.cc:695] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:09.927325 2442 consensus_queue.cc:237] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.930541 2645 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:09.931227 2645 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.925276 1628 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:09.913370 1657 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:09.934391 1628 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:09.909855 1660 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:09.909991 2837 raft_consensus.cc:3054] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:09.929651 1702 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:09.910825 1581 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
W20250114 02:54:09.939910 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 74244.1us
I20250114 02:54:09.937590 1482 catalog_manager.cc:5526] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:09.916622 2836 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:09.942284 1702 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:09.922803 1654 raft_consensus.cc:3054] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.944967 1628 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:09.946283 1702 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:09.954142 1702 leader_election.cc:304] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:09.950834 1482 catalog_manager.cc:5526] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 3 to 4, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 4 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:09.959067 1628 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:09.936115 2874 raft_consensus.cc:2798] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:09.944324 2508 raft_consensus.cc:2743] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:09.960593 2874 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:09.960418 2806 consensus_queue.cc:1046] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 02:54:09.961124 2874 raft_consensus.cc:3054] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.961669 2806 raft_consensus.cc:3049] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:09.962086 2806 raft_consensus.cc:738] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:09.962755 2756 raft_consensus.cc:2798] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:09.963361 2756 raft_consensus.cc:695] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:09.964402 2756 consensus_queue.cc:237] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.967588 2885 raft_consensus.cc:2798] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:09.968104 2885 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:09.940783 2884 raft_consensus.cc:2798] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:09.968578 2885 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.968637 2884 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:09.970917 1702 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:09.941530 2512 raft_consensus.cc:3054] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.982372 1553 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:09.982720 1628 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:09.983145 1553 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:09.975307 2645 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:09.945092 2836 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:09.976809 1656 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:09.940629 2841 raft_consensus.cc:2798] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:09.980275 1702 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:09.951614 2888 raft_consensus.cc:2798] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:09.985934 2728 raft_consensus.cc:2798] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:10.003758 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:10.004359 1733 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:09.958171 1482 catalog_manager.cc:5526] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 2 to 3, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 3 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:10.008265 1482 catalog_manager.cc:5526] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 2 to 3, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 3 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:10.012789 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:09.998827 1702 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:10.013659 1482 catalog_manager.cc:5526] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:10.015478 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.016198 1658 raft_consensus.cc:2363] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4.
I20250114 02:54:10.017935 1702 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:09.977519 2871 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:10.020176 1553 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:10.020761 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:10.021212 1702 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:10.029947 1729 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:10.030529 1702 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:10.031657 1702 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:10.032337 1482 catalog_manager.cc:5526] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 1 to 4, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 4 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:10.042866 1736 raft_consensus.cc:1270] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:10.043135 1731 raft_consensus.cc:1235] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:09.982021 2884 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:10.032795 1702 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:09.949774 2710 consensus_queue.cc:1046] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.072s
I20250114 02:54:09.986802 2508 raft_consensus.cc:2798] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:09.987357 2830 raft_consensus.cc:2798] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:09.919574 2757 consensus_queue.cc:260] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:09.986238 2730 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:10.035586 1553 leader_election.cc:400] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4.
I20250114 02:54:10.064831 1553 leader_election.cc:403] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250114 02:54:10.035416 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:09.986940 2867 raft_consensus.cc:2743] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:10.081606 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.039419 1734 raft_consensus.cc:1270] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:10.040055 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:10.123538 1728 raft_consensus.cc:2463] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:54:09.995157 2804 raft_consensus.cc:2798] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:10.027935 2868 raft_consensus.cc:2743] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:10.038213 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:10.045082 1482 catalog_manager.cc:5526] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 2 to 3, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 3 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:10.130417 1482 catalog_manager.cc:5526] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 3 to 4, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 4 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:10.065001 2442 raft_consensus.cc:2743] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:10.132671 1482 catalog_manager.cc:5526] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 3 to 4, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 4 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:10.015885 2888 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:10.020076 2645 raft_consensus.cc:2743] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:10.057606 1735 raft_consensus.cc:1235] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 02:54:10.057754 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:10.019728 2385 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:10.019095 2841 raft_consensus.cc:695] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:10.028987 2732 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:10.057870 1702 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:10.027969 2871 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:10.037485 2870 raft_consensus.cc:2743] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:10.054474 2878 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:10.049508 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.066851 1732 raft_consensus.cc:2388] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:10.102293 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:10.066787 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:10.062991 2830 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:10.105989 2719 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:10.095686 2887 raft_consensus.cc:3054] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:10.173974 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.174806 1582 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:10.111232 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:10.123029 2728 raft_consensus.cc:695] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:10.043784 2875 raft_consensus.cc:2743] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:10.097069 2881 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:10.047446 2806 consensus_queue.cc:260] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.135015 1482 catalog_manager.cc:5526] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 2 to 3, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 3 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
W20250114 02:54:10.136090 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 110020us
W20250114 02:54:10.151957 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 94163.4us
I20250114 02:54:10.018816 2838 consensus_queue.cc:1046] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20250114 02:54:10.191581 1702 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:10.141376 2871 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:10.134474 2804 raft_consensus.cc:695] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:10.141757 2878 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:10.138962 2888 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:10.173207 2830 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:10.192162 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.172706 2645 raft_consensus.cc:2743] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:10.198360 1628 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:10.201113 1702 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:10.179423 2719 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:10.219192 2904 log.cc:926] Time spent T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20250114 02:54:10.215196 2512 raft_consensus.cc:2743] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:10.211755 2870 raft_consensus.cc:2798] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:10.196981 2878 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:10.229528 1702 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
W20250114 02:54:10.219756 2903 log.cc:926] Time spent T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
I20250114 02:54:10.231825 1702 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:10.233466 1702 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:10.227041 1582 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:10.240106 1730 raft_consensus.cc:1270] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 02:54:10.195119 2874 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.197976 1654 raft_consensus.cc:2463] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:10.252581 1702 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:10.254812 1582 raft_consensus.cc:1235] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:10.256222 1733 raft_consensus.cc:1235] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 02:54:10.246757 1736 raft_consensus.cc:2463] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:10.225694 2688 raft_consensus.cc:2743] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:10.228096 1628 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:10.231323 2442 raft_consensus.cc:2758] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250114 02:54:10.286171 1658 raft_consensus.cc:2388] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:10.242718 2796 raft_consensus.cc:3049] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Stepping down as leader of term 3
W20250114 02:54:10.288190 1553 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.106.67:42283 timed out after 1.674s (SENT) [suppressed 10 similar messages]
I20250114 02:54:10.288779 1585 raft_consensus.cc:1270] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:10.290218 1585 raft_consensus.cc:1235] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 02:54:10.203445 2841 consensus_queue.cc:237] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.225688 2508 raft_consensus.cc:695] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
W20250114 02:54:10.321257 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 93255.5us
I20250114 02:54:10.258774 2870 raft_consensus.cc:695] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:10.261662 2512 raft_consensus.cc:2798] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:10.252154 2833 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:10.253242 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:54:10.359046 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 106550us
I20250114 02:54:10.359898 1702 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:10.360996 1702 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:10.317000 2868 raft_consensus.cc:2798] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:10.293599 2796 raft_consensus.cc:738] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
W20250114 02:54:10.390759 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 54222.6us
I20250114 02:54:10.232003 2728 consensus_queue.cc:237] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.334318 2655 raft_consensus.cc:2798] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:10.262076 2804 consensus_queue.cc:237] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.362046 2442 raft_consensus.cc:2743] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:10.360793 2645 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:10.361527 2730 raft_consensus.cc:695] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:10.389389 2868 raft_consensus.cc:491] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:10.355800 2732 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
W20250114 02:54:10.444643 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:54:10.361748 2867 consensus_queue.cc:1035] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20250114 02:54:10.416189 2655 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:10.426102 2917 raft_consensus.cc:2743] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250114 02:54:10.461987 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 173824us
I20250114 02:54:10.463495 1553 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:10.440886 2868 raft_consensus.cc:3054] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:10.362886 2833 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.447600 2732 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:10.461675 2655 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:10.464187 2881 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:10.459803 2870 consensus_queue.cc:237] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:10.483044 2918 log.cc:926] Time spent T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20250114 02:54:10.466948 2872 raft_consensus.cc:2798] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:10.446457 2508 consensus_queue.cc:237] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.451504 2796 consensus_queue.cc:260] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.485448 2874 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:10.534932 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 120858us
I20250114 02:54:10.540956 1628 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
W20250114 02:54:10.549834 2836 consensus_meta.cc:287] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.530s user 0.011s sys 0.000s
I20250114 02:54:10.550240 2836 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.551142 2887 raft_consensus.cc:2743] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4.
W20250114 02:54:10.533924 2929 log.cc:926] Time spent T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20250114 02:54:10.506111 2719 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:10.510684 2881 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.557461 2881 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:10.535553 2872 raft_consensus.cc:695] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:10.552202 2836 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:10.566023 1581 raft_consensus.cc:2463] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
W20250114 02:54:10.581804 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 220879us
I20250114 02:54:10.584204 1553 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:10.564755 2796 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:10.587934 1553 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:10.593482 1553 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
W20250114 02:54:10.582149 2884 consensus_meta.cc:287] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.516s user 0.010s sys 0.000s
I20250114 02:54:10.592944 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 636) took 1103 ms. Trace:
I20250114 02:54:10.479172 2730 consensus_queue.cc:237] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.594643 2926 raft_consensus.cc:2798] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:10.597295 1553 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
W20250114 02:54:10.591832 2885 consensus_meta.cc:287] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.623s user 0.003s sys 0.008s
I20250114 02:54:10.610491 1553 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:10.607951 2887 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:10.613067 2887 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:10.612792 2872 consensus_queue.cc:237] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.594344 2811 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:10.598124 2442 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
W20250114 02:54:10.617034 2837 consensus_meta.cc:287] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.675s user 0.009s sys 0.000s
I20250114 02:54:10.617468 2837 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.600450 2884 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.620245 2830 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.621601 1735 raft_consensus.cc:1235] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20250114 02:54:10.613481 2887 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:10.594163 2643 raft_consensus.cc:2798] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:54:10.620744 2884 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:10.594646 2923 raft_consensus.cc:2798] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:10.622339 2881 raft_consensus.cc:2743] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:10.635934 2843 raft_consensus.cc:2743] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:10.611239 2926 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:10.667619 1579 raft_consensus.cc:1235] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20250114 02:54:10.668242 2913 raft_consensus.cc:3049] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:10.668737 2913 raft_consensus.cc:738] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:10.669972 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.669667 2913 consensus_queue.cc:260] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.679970 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:10.672441 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
W20250114 02:54:10.673275 2932 log.cc:926] Time spent T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20250114 02:54:10.693742 2884 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:10.673458 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:10.695463 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:10.696457 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:10.697167 1579 raft_consensus.cc:2388] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 2: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:10.697841 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.698648 2645 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:10.675308 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:10.699079 2645 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:10.699105 1655 raft_consensus.cc:1235] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:10.699653 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:10.700650 1655 raft_consensus.cc:1235] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250114 02:54:10.675978 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.676577 1553 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:10.678450 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:10.701737 1656 raft_consensus.cc:2388] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:10.682262 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.683000 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.702301 2512 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:10.702332 2881 raft_consensus.cc:2743] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:10.702854 2512 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:10.694299 2884 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.704890 2884 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:10.702417 1585 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:10.707156 2875 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:10.707650 2875 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.641023 2803 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:10.709650 2875 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:10.709926 2803 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.711030 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.711305 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.711834 2803 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:10.712056 1656 raft_consensus.cc:2388] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:10.712416 2936 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:10.712867 2936 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.714314 2710 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:10.714785 2710 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.715315 2836 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:10.715806 2836 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.716542 2913 raft_consensus.cc:3054] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:10.635193 2885 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.641216 2643 raft_consensus.cc:695] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:10.595319 1584 rpcz_store.cc:276] 0114 02:54:09.489948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:09.925155 (+435207us) service_pool.cc:225] Handling call
0114 02:54:09.985636 (+ 60481us) spinlock_profiling.cc:243] Waited 6.07 ms on lock 0x7b5800000398. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce315e991 00007fbce315e842 00007fbcebe2fcd9 00007fbceafbe750 00007fbceb0847fb 00007fbceb084881 00007fbceb0482a0 00007fbceb0337e9 00007fbceb033292 00007fbce46ff694
0114 02:54:09.985795 (+ 159us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:09.985971 (+ 176us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 02:54:10.335610 (+349639us) log.cc:844] Serialized 14 byte log entry
0114 02:54:10.590693 (+255083us) log.cc:844] Serialized 14 byte log entry
0114 02:54:10.592588 (+ 1895us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:10.592635 (+ 47us) raft_consensus.cc:1640] Marking committed up to 3
0114 02:54:10.592693 (+ 58us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:10.592772 (+ 79us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:10.592886 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14815,"prepare.queue_time_us":71521,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":90,"raft.queue_time_us":16878,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":4900,"spinlock_wait_cycles":39888128,"thread_start_us":589256,"threads_started":2}
I20250114 02:54:10.717759 2643 consensus_queue.cc:237] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.691586 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:10.691787 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.692585 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
W20250114 02:54:10.732985 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 192191us
I20250114 02:54:10.634456 2878 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.718837 1661 raft_consensus.cc:1235] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:10.641772 2830 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:10.719250 2757 raft_consensus.cc:3054] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
W20250114 02:54:10.720724 2919 log.cc:926] Time spent T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.252s user 0.001s sys 0.000s
I20250114 02:54:10.721778 2442 raft_consensus.cc:2743] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:10.761946 1583 raft_consensus.cc:1270] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
W20250114 02:54:10.722888 2888 consensus_meta.cc:287] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.518s user 0.010s sys 0.000s
I20250114 02:54:10.762348 1583 pending_rounds.cc:77] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 1
I20250114 02:54:10.723676 2837 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:10.767796 2888 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.769893 2888 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:10.678016 2871 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.770833 1580 raft_consensus.cc:1235] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:10.771905 2871 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:10.772398 2806 raft_consensus.cc:3054] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:10.772678 1733 raft_consensus.cc:1235] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:10.718300 2885 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:10.729935 1585 raft_consensus.cc:2463] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:10.724920 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.691025 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.750131 2811 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:10.798101 2811 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:10.798915 1578 raft_consensus.cc:2388] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
W20250114 02:54:10.750296 2941 log.cc:926] Time spent T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20250114 02:54:10.750378 2878 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:10.640199 2923 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:10.762595 1583 pending_rounds.cc:99] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20250114 02:54:10.765082 2732 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.678571 2926 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:10.808346 1583 raft_consensus.cc:2883] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:10.817234 2923 raft_consensus.cc:3054] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:10.837745 2732 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:10.846321 1733 raft_consensus.cc:1235] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:10.848929 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:10.849982 1729 raft_consensus.cc:2388] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:10.850790 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:10.850670 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:10.851579 1733 raft_consensus.cc:2388] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:10.853842 2884 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:10.854344 2884 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:10.854732 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 121070us
I20250114 02:54:10.856232 2868 raft_consensus.cc:513] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.857419 2939 consensus_queue.cc:1035] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:10.857905 1628 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:10.858078 2868 leader_election.cc:290] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:10.857582 2874 consensus_queue.cc:1035] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:10.859095 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:10.861611 1581 raft_consensus.cc:1235] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:10.778409 2887 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.871261 2884 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:10.872407 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:10.873533 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:10.875636 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:10.877871 1735 raft_consensus.cc:2363] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4.
I20250114 02:54:10.885267 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:10.886848 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:10.892830 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:10.893149 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:10.893930 1736 raft_consensus.cc:3054] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:10.894426 1732 raft_consensus.cc:2363] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4.
I20250114 02:54:10.869386 2833 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:10.903211 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:10.904062 1733 raft_consensus.cc:2405] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:10.903966 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:10.904613 1731 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:10.904878 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:10.905377 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:54:10.906144 2930 log.cc:926] Time spent T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20250114 02:54:10.906260 1731 raft_consensus.cc:2405] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for term 5 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250114 02:54:10.906824 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.907691 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.908785 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.910491 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.910648 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.910982 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.907588 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:10.914340 1660 raft_consensus.cc:2388] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:10.917295 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.917644 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:10.918282 1578 raft_consensus.cc:2463] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:10.918519 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.919095 1655 raft_consensus.cc:3054] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 4
I20250114 02:54:10.919317 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.919703 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:10.920454 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef83c2f4c7494118a6564eaca6b7799d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.921049 1578 raft_consensus.cc:3054] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:10.921391 1656 raft_consensus.cc:1270] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20250114 02:54:10.912087 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.923086 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:10.923835 1583 raft_consensus.cc:2363] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4.
I20250114 02:54:10.874472 2949 raft_consensus.cc:2798] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:10.928803 1553 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:10.931072 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:10.932613 1736 raft_consensus.cc:2463] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:10.935518 1654 raft_consensus.cc:2363] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4.
I20250114 02:54:10.937126 1582 raft_consensus.cc:1235] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5. Ops: []
W20250114 02:54:10.937263 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 50440.1us
I20250114 02:54:10.937597 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:10.938760 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:10.939836 2950 consensus_queue.cc:1035] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:10.940701 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:10.940760 2512 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.941424 1581 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:10.942632 2891 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:10.897928 2811 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.944830 2811 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:10.930236 1553 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:10.943815 1584 raft_consensus.cc:1270] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:10.946506 1655 raft_consensus.cc:2463] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:10.881211 2645 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.948379 2710 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:10.949292 2806 raft_consensus.cc:2743] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:10.949831 2949 raft_consensus.cc:695] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:10.950687 2645 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:10.952095 1553 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:10.952381 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:10.954438 1734 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:10.956629 1702 leader_election.cc:400] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4.
I20250114 02:54:10.957171 1702 leader_election.cc:403] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250114 02:54:10.958118 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:10.962301 1702 leader_election.cc:304] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:10.965768 2943 raft_consensus.cc:2743] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4.
I20250114 02:54:10.965940 2806 raft_consensus.cc:2743] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:10.966244 2836 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:10.966475 2943 raft_consensus.cc:2798] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:10.966398 2887 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:10.967140 2936 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:10.965813 2885 consensus_queue.cc:1046] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250114 02:54:10.970274 2885 raft_consensus.cc:3049] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:10.970721 2885 raft_consensus.cc:738] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:10.971680 2885 consensus_queue.cc:260] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.973119 2885 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:10.952996 2868 consensus_queue.cc:1046] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20250114 02:54:10.974486 2936 raft_consensus.cc:3049] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:10.974855 2936 raft_consensus.cc:738] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:10.975131 1628 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:10.975628 2936 consensus_queue.cc:260] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.976732 2936 raft_consensus.cc:3054] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
W20250114 02:54:10.978420 2925 log.cc:926] Time spent T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.508s user 0.000s sys 0.003s
I20250114 02:54:10.976107 2688 raft_consensus.cc:2798] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:10.967015 2943 raft_consensus.cc:695] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:10.982363 2688 raft_consensus.cc:695] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:10.982002 2943 consensus_queue.cc:237] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.983516 2688 consensus_queue.cc:237] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:10.962535 2878 consensus_queue.cc:1035] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20250114 02:54:10.995313 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:10.996150 1736 raft_consensus.cc:2388] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:10.997942 1582 raft_consensus.cc:1270] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:10.999138 1579 raft_consensus.cc:1270] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:11.000195 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:11.000728 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:11.002038 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:11.003038 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:11.003701 1582 raft_consensus.cc:2388] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:11.005396 1734 raft_consensus.cc:1270] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:11.007164 1731 raft_consensus.cc:1235] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: []
W20250114 02:54:11.010167 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 47965us
I20250114 02:54:11.011780 1702 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.015194 1702 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.018240 1583 raft_consensus.cc:1235] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20250114 02:54:11.019002 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 54239.3us
I20250114 02:54:11.020177 1553 leader_election.cc:400] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4.
I20250114 02:54:11.020047 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:11.020910 1553 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:11.019460 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:10.928323 2732 consensus_queue.cc:1046] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.087s
I20250114 02:54:11.023753 1702 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.025425 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:11.034179 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:11.035179 1584 raft_consensus.cc:2388] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:11.024672 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:11.036839 1735 raft_consensus.cc:2388] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:11.024333 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:11.031304 2887 raft_consensus.cc:2743] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:11.052305 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:11.023957 2836 raft_consensus.cc:2743] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:11.052917 2923 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.055192 2891 raft_consensus.cc:2743] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:11.058800 2864 raft_consensus.cc:2743] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:11.063341 1732 raft_consensus.cc:1270] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:11.061714 1702 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.064826 1733 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:11.067217 1628 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.070813 1585 raft_consensus.cc:1235] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:11.072764 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:11.073449 1583 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:11.073866 1702 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.022148 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:11.022532 2645 raft_consensus.cc:2743] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:11.080104 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:10.950817 2949 consensus_queue.cc:237] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.103446 1628 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.104161 1702 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.031898 1578 raft_consensus.cc:2463] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:11.105309 1702 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.037952 2512 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:11.113279 1702 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.099387 2732 raft_consensus.cc:3054] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:11.124116 1586 raft_consensus.cc:1235] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:11.122637 1702 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:11.152388 1702 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.153470 1584 raft_consensus.cc:1235] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 5. Current term is 6. Ops: []
I20250114 02:54:11.122700 1730 raft_consensus.cc:1270] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:11.123684 2645 raft_consensus.cc:2743] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:11.110162 2874 raft_consensus.cc:2798] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:11.111963 2864 raft_consensus.cc:2743] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:11.079433 2888 consensus_queue.cc:1046] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.055s
I20250114 02:54:11.123055 2838 raft_consensus.cc:2798] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:11.108484 2868 raft_consensus.cc:2743] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:11.211035 2512 raft_consensus.cc:2798] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:11.080703 2926 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
F20250114 02:54:11.221163 425 test_workload.cc:382] Timed out: Timed out waiting for Table Creation
*** Check failure stack trace: ***
*** Aborted at 1736823251 (unix time) try "date -d @1736823251" if you are using GNU date ***
W20250114 02:54:11.224166 2933 log.cc:926] Time spent T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20250114 02:54:11.216742 2871 consensus_queue.cc:1046] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.008s
I20250114 02:54:11.439078 2871 raft_consensus.cc:3049] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:11.439455 2871 raft_consensus.cc:738] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:11.440356 2871 consensus_queue.cc:260] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.441659 2871 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:11.225013 1586 raft_consensus.cc:1235] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
W20250114 02:54:11.227212 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 73129.1us
I20250114 02:54:11.211413 2874 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:11.160368 2923 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.124823 2710 raft_consensus.cc:2743] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:11.443537 2874 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:11.443951 1702 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:11.444074 2710 consensus_queue.cc:1046] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:54:11.444865 1702 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.445101 2710 raft_consensus.cc:2798] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:11.445467 2710 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:11.445809 2710 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:11.445998 1702 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.208647 2442 raft_consensus.cc:2743] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:11.235765 2838 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:11.235908 1653 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:11.244331 1585 raft_consensus.cc:1235] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:11.245167 2926 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.245297 2887 raft_consensus.cc:3049] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:11.451880 2442 raft_consensus.cc:3049] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:11.452129 2887 raft_consensus.cc:738] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:11.452389 2796 raft_consensus.cc:2743] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:11.250348 1731 raft_consensus.cc:2405] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250114 02:54:11.238540 2916 consensus_queue.cc:1035] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20250114 02:54:11.255478 2512 raft_consensus.cc:695] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:11.451287 2878 raft_consensus.cc:3049] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:11.455833 2878 raft_consensus.cc:738] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:11.453114 2887 consensus_queue.cc:260] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.456183 2512 consensus_queue.cc:237] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.456705 2878 consensus_queue.cc:260] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.457343 2887 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:11.457679 2878 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
W20250114 02:54:11.294070 1485 rpcz_store.cc:267] Call kudu.master.MasterService.IsCreateTableDone from 127.0.0.1:39758 (request call id 44) took 111 ms (client timeout 100 ms). Trace:
I20250114 02:54:11.453672 2838 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
W20250114 02:54:11.280025 2913 consensus_meta.cc:287] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.563s user 0.006s sys 0.005s
W20250114 02:54:11.458737 1485 rpcz_store.cc:269] 0114 02:54:11.155447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:11.212427 (+ 56980us) service_pool.cc:225] Handling call
0114 02:54:11.236646 (+ 24219us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:11.236887 (+ 241us) catalog_manager.cc:2246] Verify if the table creation is in progress for test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]
0114 02:54:11.239208 (+ 2321us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250114 02:54:11.312994 2953 log.cc:926] Time spent T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20250114 02:54:11.333005 2655 consensus_meta.cc:287] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.847s user 0.007s sys 0.003s
I20250114 02:54:11.335104 2732 raft_consensus.cc:2743] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:54:11.351591 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 248228us
I20250114 02:54:11.454697 2936 raft_consensus.cc:2743] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250114 02:54:11.392999 2957 log.cc:926] Time spent T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20250114 02:54:11.462513 2972 consensus_queue.cc:1035] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:11.381605 2971 raft_consensus.cc:491] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.463475 2971 raft_consensus.cc:513] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.465585 2971 leader_election.cc:290] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.415306 1580 raft_consensus.cc:1270] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:11.452384 2442 raft_consensus.cc:738] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:11.467949 2442 consensus_queue.cc:260] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.468883 2442 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:11.468796 1628 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.470180 1628 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:11.470713 2513 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.470654 2874 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.470753 2867 consensus_queue.cc:1046] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:11.471343 2513 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.472182 2867 consensus_queue.cc:1035] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:54:11.472560 2874 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.473158 2513 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:11.323133 2954 log.cc:926] Time spent T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20250114 02:54:11.460458 2655 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.476049 2655 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.476724 1659 raft_consensus.cc:1235] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
W20250114 02:54:11.352725 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 252465us
I20250114 02:54:11.477370 2838 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.454668 1702 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.478832 1702 leader_election.cc:400] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4.
I20250114 02:54:11.479522 1702 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.479642 1553 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
W20250114 02:54:11.477794 2948 log.cc:926] Time spent T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.368s user 0.000s sys 0.005s
I20250114 02:54:11.480973 1702 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:11.470894 2949 raft_consensus.cc:2798] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:54:11.481878 1661 raft_consensus.cc:1235] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:11.473122 2874 raft_consensus.cc:3049] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:11.483879 1654 raft_consensus.cc:1270] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:11.484833 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:11.484926 2949 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:11.484941 2732 raft_consensus.cc:2743] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:11.487756 2949 raft_consensus.cc:3054] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:11.485534 1658 raft_consensus.cc:1270] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:54:11.489495 2796 raft_consensus.cc:2798] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:11.489876 2796 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:11.490221 2796 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:11.490844 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:11.486297 2923 raft_consensus.cc:491] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:11.492172 2923 raft_consensus.cc:513] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.493988 2923 leader_election.cc:290] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.494849 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:11.495642 2838 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.498201 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:11.498466 1585 raft_consensus.cc:1270] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 02:54:11.498795 1660 raft_consensus.cc:2463] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:11.500349 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:11.501601 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:11.502187 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:11.502650 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:11.504741 1658 raft_consensus.cc:1270] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250114 02:54:11.505112 1658 pending_rounds.cc:77] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Aborting all ops after (but not including) 1
I20250114 02:54:11.505345 1658 pending_rounds.cc:99] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 02:54:11.505558 1658 raft_consensus.cc:2883] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:11.508332 1585 raft_consensus.cc:1235] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:11.508483 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:11.509143 2719 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.509472 1656 raft_consensus.cc:1235] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:11.509637 2974 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:11.486771 2936 raft_consensus.cc:2743] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:11.509869 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:11.494860 2916 consensus_queue.cc:1035] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20250114 02:54:11.511149 1657 raft_consensus.cc:1270] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:11.516081 1661 raft_consensus.cc:1270] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:11.518019 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:11.518056 1553 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:11.499986 2732 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.523581 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:11.524235 1729 raft_consensus.cc:2388] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:11.526410 1729 raft_consensus.cc:1270] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:11.487305 2874 raft_consensus.cc:738] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:11.527776 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:11.528795 1553 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:11.529345 2804 raft_consensus.cc:2743] T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:11.530176 1736 raft_consensus.cc:1235] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:11.531045 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:11.509578 2719 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.511000 2923 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:11.534859 2913 consensus_queue.cc:1035] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:11.540006 1553 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
W20250114 02:54:11.535796 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:54:11.536631 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:11.510196 2974 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.506134 2912 raft_consensus.cc:2743] T d43434bae9eb4b2fb3699343149bb385 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:11.543465 1553 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:11.546854 1553 leader_election.cc:400] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4.
I20250114 02:54:11.547513 1553 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:11.548573 1661 raft_consensus.cc:1270] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:11.551167 1702 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.552052 1553 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:11.543095 2913 consensus_queue.cc:1225] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer b89550620e1d455d887be72232bb5454 log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250114 02:54:11.554361 1553 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:11.520560 2513 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:11.544390 2843 raft_consensus.cc:2743] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:11.593036 1735 raft_consensus.cc:1270] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:54:11.594098 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43434bae9eb4b2fb3699343149bb385" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:11.537506 2732 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.597620 1580 raft_consensus.cc:1270] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:11.592696 2968 raft_consensus.cc:2798] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:11.544034 2811 raft_consensus.cc:2798] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:11.599287 2655 consensus_queue.cc:1046] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:11.602135 2688 consensus_queue.cc:1046] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:11.602308 2884 consensus_queue.cc:1046] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:11.601737 2936 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.605212 2949 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.861260 2804 raft_consensus.cc:2743] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:11.862913 2949 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.605950 2926 raft_consensus.cc:2798] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:11.861296 2936 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.513859 2972 consensus_queue.cc:1046] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20250114 02:54:11.864161 1662 raft_consensus.cc:1235] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 1. Current term is 2. Ops: []
W20250114 02:54:11.606935 2986 log.cc:926] Time spent T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20250114 02:54:11.865492 2936 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:11.611243 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 60176.2us
I20250114 02:54:11.612818 2796 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.591252 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:11.866884 1736 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:11.867193 1702 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.867586 2796 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:11.868132 2796 raft_consensus.cc:2743] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:11.620095 2710 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.869554 2916 consensus_queue.cc:1035] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:11.612188 2871 raft_consensus.cc:2743] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:11.870154 2710 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:11.851810 2512 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.870934 2512 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.871838 2968 raft_consensus.cc:695] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:11.872208 2888 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.872833 2512 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:11.872965 2872 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:11.872710 2968 consensus_queue.cc:237] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.873520 2891 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.873435 2872 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.873574 2864 consensus_queue.cc:1035] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 02:54:11.874130 2891 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.875046 2804 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:11.853868 2719 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.875361 2872 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.875546 2871 consensus_queue.cc:1035] T 7cd3f8af715b4ed8a97965a52b089a21 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:11.876080 2891 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.631587 2923 raft_consensus.cc:695] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:11.856992 2878 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.655761 2811 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:11.875563 2804 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.634497 2513 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.877420 2878 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.878666 2513 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.877444 2811 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:11.879536 2878 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:11.881536 1702 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.881904 2878 consensus_queue.cc:1035] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:11.882218 2512 raft_consensus.cc:2743] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:11.882670 2804 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.883083 1702 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:11.880791 2923 consensus_queue.cc:237] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.623203 2803 raft_consensus.cc:2743] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250114 02:54:11.860071 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 305786us
I20250114 02:54:11.858428 2732 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:11.863559 2926 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:11.889168 2926 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:11.890071 2891 consensus_queue.cc:1035] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:54:11.892477 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:11.892547 1553 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:11.536165 2868 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.894822 2868 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.669611 2874 consensus_queue.cc:260] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.851922 2442 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:11.874917 2864 consensus_queue.cc:1035] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:11.883749 2878 raft_consensus.cc:2743] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:11.895473 2887 consensus_queue.cc:1046] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.003s
I20250114 02:54:11.872638 2888 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.895797 2891 raft_consensus.cc:2743] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 02:54:11.896706 2868 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:11.896993 2871 consensus_queue.cc:1035] T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20250114 02:54:11.895884 2442 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.900180 2888 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.899940 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:11.897925 2926 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.897434 2874 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:11.901827 2442 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:11.902477 2926 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.885775 2811 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.904875 2843 raft_consensus.cc:3049] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:11.905251 2843 raft_consensus.cc:738] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:11.912150 2843 consensus_queue.cc:260] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.913341 2843 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
W20250114 02:54:11.919095 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 291612us
I20250114 02:54:11.919987 2643 consensus_queue.cc:1035] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:11.920675 2880 consensus_queue.cc:1046] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 02:54:11.921123 1628 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:11.921674 2643 consensus_queue.cc:1035] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:11.922035 2688 consensus_queue.cc:1225] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer b89550620e1d455d887be72232bb5454 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 02:54:11.922345 2884 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:11.922650 2688 raft_consensus.cc:3049] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:11.922781 2884 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:11.923141 2688 raft_consensus.cc:738] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:11.923398 2913 raft_consensus.cc:3049] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:11.923769 2913 raft_consensus.cc:738] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:11.923218 2884 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:11.924114 2688 consensus_queue.cc:260] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.924472 2913 consensus_queue.cc:260] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.925537 2688 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:11.925968 2972 raft_consensus.cc:3049] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:11.926395 2972 raft_consensus.cc:738] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:11.927062 2655 raft_consensus.cc:3049] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:11.927517 2655 raft_consensus.cc:738] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:11.927268 2972 consensus_queue.cc:260] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.928272 2655 consensus_queue.cc:260] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:11.928763 2972 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:11.929425 2655 raft_consensus.cc:3054] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:11.941062 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:11.946388 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:11.947101 1585 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:11.948827 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:11.949460 1585 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:11.921746 2880 raft_consensus.cc:3049] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:11.953778 2880 raft_consensus.cc:738] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:11.957026 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:11.957561 1662 raft_consensus.cc:3054] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:11.961594 1736 raft_consensus.cc:1235] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:11.962055 2913 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:11.963727 1662 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:11.961426 2972 consensus_queue.cc:1035] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:11.969614 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:11.970842 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:11.976329 1662 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:11.860652 2974 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.979846 1731 raft_consensus.cc:1270] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 02:54:11.980613 1662 raft_consensus.cc:2463] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:11.985036 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:11.981986 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:11.985986 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:11.986537 1587 raft_consensus.cc:3054] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:11.987063 1729 raft_consensus.cc:1235] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:11.989233 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:11.989387 1736 raft_consensus.cc:1270] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:54:11.989406 1735 raft_consensus.cc:1270] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:11.990098 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:11.991037 1583 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:11.982663 1585 raft_consensus.cc:2463] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:11.992756 1728 raft_consensus.cc:1270] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:11.993217 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:11.992785 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:11.975474 2811 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:11.998914 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.011346 1628 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:11.973811 2887 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.012833 1628 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:11.983057 1580 raft_consensus.cc:1270] T ef83c2f4c7494118a6564eaca6b7799d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:11.999300 1702 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:12.015275 1702 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:12.015719 1628 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:12.017105 1702 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:12.023720 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.025254 1702 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:12.027508 1735 pending_rounds.cc:77] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 0
I20250114 02:54:12.027858 1735 pending_rounds.cc:99] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250114 02:54:12.028106 1735 raft_consensus.cc:2883] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:11.995762 2442 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:12.034610 1658 raft_consensus.cc:1270] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250114 02:54:12.035019 1658 pending_rounds.cc:77] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Aborting all ops after (but not including) 2
I20250114 02:54:12.035264 1658 pending_rounds.cc:99] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250114 02:54:12.035513 1658 raft_consensus.cc:2883] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:11.989739 2732 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:11.995571 2868 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.007048 2867 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.032373 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:12.045571 2867 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.047876 2867 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.029222 2887 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.049868 2887 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.033070 2442 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.029903 2871 raft_consensus.cc:2798] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:12.031570 2885 raft_consensus.cc:2743] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:12.030965 2756 raft_consensus.cc:2743] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:12.030803 2796 raft_consensus.cc:2798] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:12.043236 2943 raft_consensus.cc:2798] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:11.991242 2880 consensus_queue.cc:260] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.052138 2732 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.028656 2911 consensus_queue.cc:1035] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20250114 02:54:12.039057 2841 consensus_queue.cc:1035] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20250114 02:54:12.059791 2878 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.060465 2871 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:12.052695 2868 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.070128 2880 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:12.071699 2878 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.080343 2878 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.080549 2884 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.082120 2884 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.082756 2943 raft_consensus.cc:695] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
PC: @ 0x0 (unknown)
W20250114 02:54:12.113513 1539 catalog_manager.cc:5989] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72: Tablet 985a08c846764478aaeb97aa5c283bf9 (table test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]) was not created within the allowed timeout. Replacing with a new tablet c4ff92fc16244b87b6d30cf38194182d
I20250114 02:54:12.148061 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
*** SIGABRT (@0x3e8000001a9) received by PID 425 (TID 0x7fbcded6c880) from PID 425; stack trace: ***
I20250114 02:54:12.022619 1587 raft_consensus.cc:2463] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
@ 0x2c9170 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2cb653 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20250114 02:54:12.150542 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:12.151014 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.153004 1580 raft_consensus.cc:1270] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250114 02:54:12.148102 1585 raft_consensus.cc:1235] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:12.147635 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:12.162618 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:12.165278 1730 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:12.168975 2732 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:12.171850 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 156128us
I20250114 02:54:12.172775 2943 consensus_queue.cc:237] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.175089 2884 raft_consensus.cc:2743] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:12.165691 1702 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:12.177426 3009 raft_consensus.cc:2798] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:12.071794 2871 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:12.188776 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:12.170594 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 252546us
I20250114 02:54:12.179505 1659 raft_consensus.cc:2405] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 02:54:12.063422 2864 consensus_queue.cc:1035] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20250114 02:54:12.178000 3009 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:12.202080 1731 raft_consensus.cc:1235] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:12.212718 1581 raft_consensus.cc:2388] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:12.177474 2868 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.222478 1587 raft_consensus.cc:1270] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:12.203819 1628 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
W20250114 02:54:12.226095 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 51664.5us
I20250114 02:54:12.227305 2911 raft_consensus.cc:2743] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:12.228648 1628 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:12.230031 2911 raft_consensus.cc:2798] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:12.230480 2911 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:12.231019 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
W20250114 02:54:12.232159 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 53427us
I20250114 02:54:12.233798 1702 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:12.233686 2864 consensus_queue.cc:1035] T ef83c2f4c7494118a6564eaca6b7799d P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 02:54:12.234541 1553 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:12.236953 1702 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:12.237926 1586 raft_consensus.cc:1270] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 02:54:12.238056 2878 raft_consensus.cc:2743] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:12.238873 2887 raft_consensus.cc:2798] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:12.239334 2887 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:12.239786 2843 consensus_queue.cc:1035] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:12.236212 2923 consensus_queue.cc:1035] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20250114 02:54:12.239929 1736 raft_consensus.cc:1235] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:12.239756 2887 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:12.236013 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:12.266125 1583 raft_consensus.cc:1270] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:12.197767 2442 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.213744 3009 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:12.231062 2911 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:12.310910 2998 consensus_queue.cc:1035] T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20250114 02:54:12.318408 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 87927.8us
I20250114 02:54:12.320739 2972 consensus_queue.cc:1046] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:12.322604 2688 raft_consensus.cc:3049] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 02:54:12.323032 2688 raft_consensus.cc:738] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:12.324148 2688 consensus_queue.cc:260] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.325614 2688 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:12.327051 1539 catalog_manager.cc:5813] T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72: Sending DeleteTablet for 3 replicas of tablet 985a08c846764478aaeb97aa5c283bf9
I20250114 02:54:12.343715 2971 consensus_queue.cc:1046] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250114 02:54:12.344905 1553 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:12.345748 2998 raft_consensus.cc:2743] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:12.346001 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985a08c846764478aaeb97aa5c283bf9" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.346436 1553 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:12.346671 2796 raft_consensus.cc:695] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:12.347116 2998 raft_consensus.cc:2743] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:12.348212 1553 leader_election.cc:304] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:12.347949 2796 consensus_queue.cc:237] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.348946 2998 raft_consensus.cc:2743] T d43434bae9eb4b2fb3699343149bb385 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:12.351101 2936 raft_consensus.cc:2743] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:12.351101 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.352579 1583 raft_consensus.cc:2405] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:12.352908 1717 tablet_service.cc:1514] Processing DeleteTablet for tablet 985a08c846764478aaeb97aa5c283bf9 with delete_type TABLET_DATA_DELETED (Replaced by c4ff92fc16244b87b6d30cf38194182d at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:36426
I20250114 02:54:12.354593 2864 raft_consensus.cc:2743] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:12.368528 2756 raft_consensus.cc:3049] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:12.368896 2756 raft_consensus.cc:738] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:12.370663 2880 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:12.371256 2719 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.371299 2880 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.377545 2871 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.379141 2880 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.379653 2871 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.379814 2719 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.390782 2880 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:12.394294 2972 consensus_queue.cc:1035] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.015s
I20250114 02:54:12.408833 2880 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.382553 2719 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.393888 2971 consensus_queue.cc:1046] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.017s
I20250114 02:54:12.369848 2756 consensus_queue.cc:260] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.428357 2971 raft_consensus.cc:3049] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:12.428800 2971 raft_consensus.cc:738] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:12.430989 2841 consensus_queue.cc:1035] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 02:54:12.432695 2841 consensus_queue.cc:1046] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:12.433890 2841 consensus_queue.cc:1035] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:12.435338 2949 raft_consensus.cc:3049] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:12.435184 2838 consensus_queue.cc:1035] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:12.435804 2949 raft_consensus.cc:738] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:12.436861 2949 consensus_queue.cc:260] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:12.438967 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 196718us
I20250114 02:54:12.439615 2913 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:12.440320 2913 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.442957 2442 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:12.443588 2442 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.438355 2949 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:12.455230 1483 catalog_manager.cc:5526] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 3 to 4, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 4 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:12.456166 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.456987 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.442984 2913 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.458396 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.460477 2871 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.461055 2871 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.461720 3015 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.429948 2971 consensus_queue.cc:260] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.463097 2868 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.463485 2971 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:12.463624 2868 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.464542 2871 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.465601 2868 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.466787 2512 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:12.466755 2968 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.467600 1657 raft_consensus.cc:1235] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:12.467470 2512 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.467670 2968 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.469511 2512 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.469779 2968 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.466547 3013 raft_consensus.cc:491] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:12.457711 1483 catalog_manager.cc:5526] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 3, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:12.471457 1702 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:12.473171 2756 raft_consensus.cc:3054] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:12.462218 3015 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.451606 2887 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.486582 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.453543 2719 consensus_queue.cc:1046] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20250114 02:54:12.492720 2916 raft_consensus.cc:2743] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:12.498661 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.475872 3013 raft_consensus.cc:513] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.497387 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:12.493463 2880 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.509322 1735 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:12.513711 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:12.525405 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 54119.3us
I20250114 02:54:12.519249 1653 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
W20250114 02:54:12.531522 3021 log.cc:926] Time spent T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.091s user 0.004s sys 0.000s
@ 0x7fbce64db980 (unknown) at ??:0
I20250114 02:54:12.532490 3015 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.518175 2916 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:12.516914 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.534647 2887 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.541412 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:12.543498 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:12.549995 3025 tablet_replica.cc:331] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454: stopping tablet replica
I20250114 02:54:12.544494 2442 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.568284 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.568434 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.548194 3015 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:12.540021 2911 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.579026 1579 raft_consensus.cc:2405] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:12.580394 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.569357 2887 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:12.583379 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:12.583983 1730 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:12.589185 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 985a08c846764478aaeb97aa5c283bf9 with delete_type TABLET_DATA_DELETED (Replaced by c4ff92fc16244b87b6d30cf38194182d at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:48630
I20250114 02:54:12.589453 1628 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:12.589887 2972 consensus_queue.cc:1046] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:12.591023 1732 raft_consensus.cc:1235] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:12.591128 2972 consensus_queue.cc:1035] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:12.592656 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.589243 2843 raft_consensus.cc:3049] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:54:12.571852 2719 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:12.595795 1583 raft_consensus.cc:2388] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:12.592554 2867 raft_consensus.cc:3049] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:12.601522 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:12.601558 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.593698 2843 raft_consensus.cc:738] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:12.603075 1483 catalog_manager.cc:5526] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 2 to 3, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 3 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:12.586452 2923 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:12.582854 2887 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.636250 1662 raft_consensus.cc:1235] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:12.658080 2688 consensus_queue.cc:1046] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:12.658672 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.660369 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.658134 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:12.660815 1483 catalog_manager.cc:5526] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 3 to 5, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 5 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:12.663154 3025 raft_consensus.cc:2238] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Raft consensus shutting down.
I20250114 02:54:12.663800 1483 catalog_manager.cc:5526] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd reported cstate change: term changed from 1 to 4, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 4 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:12.664968 1579 raft_consensus.cc:1270] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:12.665761 2923 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.666875 2710 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:12.635497 3022 log.cc:926] Time spent T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.166s user 0.003s sys 0.000s
I20250114 02:54:12.684363 1735 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:12.636978 1578 raft_consensus.cc:2388] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:12.576594 2916 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.607358 3015 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.691632 2887 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.692023 2916 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:12.635232 3030 log.cc:926] Time spent T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20250114 02:54:12.692699 3015 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.667547 2710 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.694880 2710 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.685837 2884 raft_consensus.cc:2743] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:12.698699 2804 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.699244 2804 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.701051 2804 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.703557 2911 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.603886 2867 raft_consensus.cc:738] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:12.711694 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:12.713069 1579 raft_consensus.cc:2388] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
W20250114 02:54:12.713675 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 55767.1us
I20250114 02:54:12.636497 2719 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.719671 2719 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.721621 3013 leader_election.cc:290] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.667793 2923 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.724443 2710 consensus_queue.cc:1035] T 6b73d9b5d01245f8be76a02efe7e5eca P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:12.726238 3025 raft_consensus.cc:2267] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:12.734664 1702 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:12.603703 3009 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:12.729095 3031 log.cc:926] Time spent T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.222s user 0.000s sys 0.003s
I20250114 02:54:12.729507 1660 raft_consensus.cc:2463] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:12.731287 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:12.731282 2926 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
W20250114 02:54:12.731348 3028 log.cc:926] Time spent T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20250114 02:54:12.733310 1580 raft_consensus.cc:1270] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250114 02:54:12.693781 2843 consensus_queue.cc:260] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.750684 1653 raft_consensus.cc:2363] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6.
I20250114 02:54:12.759649 1628 leader_election.cc:304] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:12.762598 1729 raft_consensus.cc:1235] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 02:54:12.762822 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.763795 2923 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:12.763993 1659 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:12.764362 2923 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.766419 2923 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.761219 2841 consensus_queue.cc:1035] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20250114 02:54:12.772500 2841 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.773039 2841 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.773655 2867 consensus_queue.cc:260] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.775434 2867 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:12.751178 3013 raft_consensus.cc:2743] T cd7d01e627ea4bfc86adf4c4fa3f6139 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:12.775700 3009 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.781513 3025 ts_tablet_manager.cc:1905] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:12.784376 2926 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:12.802481 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 67943.3us
I20250114 02:54:12.832262 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.831699 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.836953 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.835355 1662 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:12.840755 1658 raft_consensus.cc:2388] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:12.830324 2971 raft_consensus.cc:2743] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:12.829620 2843 raft_consensus.cc:3054] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:54:12.851680 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 92154.6us
I20250114 02:54:12.835352 2867 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.852875 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.835531 3009 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.855789 2688 consensus_queue.cc:1035] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:12.858922 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.861034 1661 raft_consensus.cc:1235] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 5. Current term is 6. Ops: []
I20250114 02:54:12.862147 2926 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.873128 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.867926 1717 tablet_service.cc:1467] Processing CreateTablet for tablet c4ff92fc16244b87b6d30cf38194182d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250114 02:54:12.874846 1717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4ff92fc16244b87b6d30cf38194182d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.869717 3009 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.877471 3009 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:12.870736 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.878808 2804 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:12.871596 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.879509 2804 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.880012 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.881189 2926 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:12.882061 2968 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:12.871346 2867 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:12.864058 2985 log.cc:926] Time spent T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20250114 02:54:12.882740 2968 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.887444 1628 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:12.887699 2971 consensus_queue.cc:1035] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:12.887015 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.889158 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.887045 1570 tablet_service.cc:1514] Processing DeleteTablet for tablet 985a08c846764478aaeb97aa5c283bf9 with delete_type TABLET_DATA_DELETED (Replaced by c4ff92fc16244b87b6d30cf38194182d at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:34912
I20250114 02:54:12.890817 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.891523 1584 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:12.892354 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.890314 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.889024 2912 raft_consensus.cc:2743] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:12.886217 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.900853 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.900949 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.902386 1653 raft_consensus.cc:2388] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:12.904930 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.906760 2911 raft_consensus.cc:3049] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:12.906708 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:12.907236 2911 raft_consensus.cc:738] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:12.907478 1728 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:12.907625 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:12.907781 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:12.908298 1729 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:12.908463 1727 raft_consensus.cc:2388] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:12.908058 2911 consensus_queue.cc:260] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.908861 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:12.909003 1586 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:12.909113 2911 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:12.906050 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.909790 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:12.910022 1579 raft_consensus.cc:2463] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:12.910497 1586 raft_consensus.cc:2388] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:12.909235 1581 raft_consensus.cc:1235] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250114 02:54:12.864753 2841 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.881781 2926 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.882844 2804 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.924443 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:12.934072 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:12.938393 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:12.939217 1736 raft_consensus.cc:2388] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:12.946931 1660 raft_consensus.cc:1270] T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:12.947252 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:12.948400 3049 tablet_replica.cc:331] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd: stopping tablet replica
I20250114 02:54:12.949049 3039 tablet_replica.cc:331] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: stopping tablet replica
I20250114 02:54:12.951341 3039 raft_consensus.cc:2238] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:12.952118 3039 raft_consensus.cc:2267] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:12.957665 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:12.957932 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:12.959049 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:12.959276 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:12.959928 1734 raft_consensus.cc:2463] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:12.959993 1729 raft_consensus.cc:2388] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:12.960359 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:12.961607 2885 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.962172 2885 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:12.964632 2885 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.979676 2968 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:12.983433 2867 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:12.994313 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 60536.7us
I20250114 02:54:12.999612 1654 raft_consensus.cc:1270] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:13.001822 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.007805 1628 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
W20250114 02:54:13.009963 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:54:13.002803 2887 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.013459 2887 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.014003 2926 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.014866 3039 ts_tablet_manager.cc:1905] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:13.016999 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.018035 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.010798 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.023255 1702 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:13.023663 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.024473 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.024492 1735 raft_consensus.cc:2388] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:13.023303 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:13.029801 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.029795 1702 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:13.032347 1731 raft_consensus.cc:2388] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:13.002547 2867 raft_consensus.cc:491] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:13.033339 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.034061 1735 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:13.035678 2887 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.036818 1577 tablet_service.cc:1467] Processing CreateTablet for tablet c4ff92fc16244b87b6d30cf38194182d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250114 02:54:13.037482 1553 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:13.038321 1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4ff92fc16244b87b6d30cf38194182d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.038046 2887 consensus_queue.cc:1046] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:13.037170 1732 raft_consensus.cc:1270] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:13.042196 1654 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:13.043016 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:13.045001 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.021494 2878 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.049117 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.049783 1653 raft_consensus.cc:2463] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:13.041821 2719 raft_consensus.cc:3049] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:13.078006 1732 raft_consensus.cc:1270] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:13.039152 2887 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:13.045760 2841 raft_consensus.cc:2798] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:13.091419 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.081779 2878 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.095515 2878 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:13.096475 3043 log.cc:926] Time spent T 8d4d46461bf745639ea4f64fa3bcb996 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20250114 02:54:13.085551 2868 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.097231 2868 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.098264 3049 raft_consensus.cc:2238] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:13.099102 2868 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.087977 2916 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:13.099972 2916 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.101781 2916 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.102227 2732 consensus_queue.cc:1046] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:13.103082 2732 raft_consensus.cc:3049] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 02:54:13.103397 2732 raft_consensus.cc:738] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:13.104218 2732 consensus_queue.cc:260] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.105332 2732 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:13.046432 2867 raft_consensus.cc:513] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.107489 3049 raft_consensus.cc:2267] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:13.108047 2968 consensus_queue.cc:1046] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 02:54:13.111640 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.113476 1587 raft_consensus.cc:1270] T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:13.115377 2923 raft_consensus.cc:3049] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:13.115442 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.115773 3015 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.116022 2923 raft_consensus.cc:738] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:13.116439 1584 raft_consensus.cc:2405] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250114 02:54:13.116299 3015 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.095337 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.117159 2923 consensus_queue.cc:260] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.117910 1653 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:13.118300 3015 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.122280 1653 raft_consensus.cc:1270] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:13.125768 1653 raft_consensus.cc:1270] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:13.089088 2804 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.128175 2804 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.092237 1660 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:13.143919 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.144429 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.144657 1654 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:13.145653 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.147639 2843 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.152889 2843 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:13.149794 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 102046us
I20250114 02:54:13.083985 2719 raft_consensus.cc:738] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:13.086931 2887 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:13.148684 2710 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.134171 2804 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.046106 2796 raft_consensus.cc:2798] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:13.047441 3009 raft_consensus.cc:2798] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:13.160722 1702 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:13.161934 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.162242 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.174576 1628 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:13.156556 2841 raft_consensus.cc:695] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:13.156791 2868 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.169879 2887 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:13.194208 1702 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:13.195318 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.179240 2796 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:13.222483 2796 raft_consensus.cc:3054] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:13.223078 2841 consensus_queue.cc:237] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.187508 3009 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:13.196732 2868 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:13.231447 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 138410us
I20250114 02:54:13.200795 2885 raft_consensus.cc:2743] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 02:54:13.200362 3015 raft_consensus.cc:2743] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:13.225227 2916 raft_consensus.cc:2743] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:13.227085 3009 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:13.199795 2710 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.244361 3013 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.248136 3013 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.252782 2923 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:13.256419 3013 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.257491 3015 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:13.261370 3042 log.cc:926] Time spent T 5d77d39f9a4b4a57ae17db8e50237cdb P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.248s user 0.003s sys 0.001s
I20250114 02:54:13.262332 3049 ts_tablet_manager.cc:1905] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:13.263692 1653 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:13.259582 3015 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.244521 2843 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.267540 3015 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.267678 1482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39748 (request call id 100) took 3535 ms. Trace:
I20250114 02:54:13.267994 2867 leader_election.cc:290] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.271759 1628 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:13.272918 2884 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:13.272616 2913 consensus_queue.cc:1046] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 02:54:13.273293 2884 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:13.273695 2884 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:13.274184 2912 raft_consensus.cc:3049] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:13.274643 2912 raft_consensus.cc:738] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:13.275187 1628 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:13.275707 2912 consensus_queue.cc:260] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.276553 1628 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:13.277302 2912 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
W20250114 02:54:13.277539 3053 log.cc:926] Time spent T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.103s user 0.000s sys 0.001s
I20250114 02:54:13.277977 2971 consensus_queue.cc:1046] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:13.278797 2688 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:13.279264 2688 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:13.279404 2971 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:13.279758 2688 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:13.279919 2971 raft_consensus.cc:695] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:13.280879 2971 consensus_queue.cc:237] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.306298 1553 leader_election.cc:400] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6.
I20250114 02:54:13.306788 1553 leader_election.cc:403] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250114 02:54:13.268090 1482 rpcz_store.cc:276] 0114 02:54:09.667055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:09.668553 (+ 1498us) service_pool.cc:225] Handling call
0114 02:54:13.197438 (+3528885us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:10.900988 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:11.105845 (+204857us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:11.109112 (+ 3267us) write_op.cc:435] Acquired schema lock
0114 02:54:11.109140 (+ 28us) tablet.cc:659] Decoding operations
0114 02:54:11.294003 (+184863us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:11.294076 (+ 73us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:11.294104 (+ 28us) tablet.cc:682] Acquiring locks for 11 operations
0114 02:54:11.336200 (+ 42096us) tablet.cc:698] Row locks acquired
0114 02:54:11.336243 (+ 43us) write_op.cc:260] PREPARE: finished
0114 02:54:11.336426 (+ 183us) write_op.cc:270] Start()
0114 02:54:11.336571 (+ 145us) write_op.cc:276] Timestamp: P: 1736823251336374 usec, L: 0
0114 02:54:11.336596 (+ 25us) op_driver.cc:348] REPLICATION: starting
0114 02:54:11.355995 (+ 19399us) log.cc:844] Serialized 4388 byte log entry
0114 02:54:11.858907 (+502912us) op_driver.cc:464] REPLICATION: finished
0114 02:54:11.892877 (+ 33970us) write_op.cc:301] APPLY: starting
0114 02:54:11.892949 (+ 72us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:11.893143 (+ 194us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:11.893244 (+ 101us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:11.897163 (+ 3919us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:11.897357 (+ 194us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0114 02:54:11.897396 (+ 39us) write_op.cc:312] APPLY: finished
0114 02:54:11.899294 (+ 1898us) log.cc:844] Serialized 82 byte log entry
0114 02:54:11.900621 (+ 1327us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:11.900838 (+ 217us) write_op.cc:454] Released schema lock
0114 02:54:11.900885 (+ 47us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:11.901227 (+ 342us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:11.901255 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":632,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":33745,"apply.run_cpu_time_us":2565,"apply.run_wall_time_us":8324,"num_ops":11,"prepare.queue_time_us":645477,"prepare.run_cpu_time_us":19167,"prepare.run_wall_time_us":525535,"raft.queue_time_us":334275,"raft.run_cpu_time_us":6798,"raft.run_wall_time_us":45070,"replication_time_us":522183,"thread_start_us":612303,"threads_started":5,"wal-append.queue_time_us":151660}]]}
I20250114 02:54:13.286002 2756 raft_consensus.cc:3049] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:13.202734 2719 consensus_queue.cc:260] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.311442 2756 raft_consensus.cc:738] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
W20250114 02:54:13.334307 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 133319us
I20250114 02:54:13.339316 1702 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
W20250114 02:54:13.347020 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 108931us
I20250114 02:54:13.350256 1702 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:13.351765 1702 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:13.357645 1553 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:13.348558 2867 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.359215 1702 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:13.360332 1702 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:13.353655 2878 raft_consensus.cc:2743] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:13.354698 2719 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:13.373769 1702 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:13.357654 3013 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:13.356669 2804 raft_consensus.cc:2743] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6.
I20250114 02:54:13.387578 3060 raft_consensus.cc:2743] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:13.388233 3060 raft_consensus.cc:2798] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:13.388563 3060 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:13.388904 3060 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:13.389771 3025 ts_tablet_manager.cc:1918] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 4.2
I20250114 02:54:13.390367 3025 log.cc:1198] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454: Deleting WAL directory at /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-2-root/wals/985a08c846764478aaeb97aa5c283bf9
I20250114 02:54:13.390889 2868 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.391397 3025 ts_tablet_manager.cc:1939] T 985a08c846764478aaeb97aa5c283bf9 P b89550620e1d455d887be72232bb5454: Deleting consensus metadata
I20250114 02:54:13.391466 2868 raft_consensus.cc:2798] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:13.391935 2868 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:13.392259 2868 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:13.393729 3009 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.394426 3025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36426 (request call id 59) took 1042 ms. Trace:
I20250114 02:54:13.395167 3057 consensus_queue.cc:1035] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 02:54:13.374866 2923 raft_consensus.cc:2798] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election won for term 4
W20250114 02:54:13.396857 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 120396us
I20250114 02:54:13.385421 2878 raft_consensus.cc:2743] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:13.361151 2867 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.394852 3025 rpcz_store.cc:276] 0114 02:54:12.351925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:12.352384 (+ 459us) service_pool.cc:225] Handling call
0114 02:54:12.452751 (+100367us) ts_tablet_manager.cc:1145] Deleting tablet 985a08c846764478aaeb97aa5c283bf9
0114 02:54:12.473802 (+ 21051us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:12.725790 (+251988us) spinlock_profiling.cc:243] Waited 60.1 ms on lock 0x7b540028f3a0. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce606366d 00007fbce60512b0 00007fbce60b4aba 00007fbce60dd375 00007fbce9aa51ac 00007fbceb082d5a 00007fbceb0b2ca0 00007fbceb099e6d 00007fbceb099e19 00007fbceb099da9
0114 02:54:13.017235 (+291445us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:13.389442 (+372207us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:13.394336 (+ 4894us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138179840,"tablet-delete.queue_time_us":99001,"thread_start_us":59357,"threads_started":1}
I20250114 02:54:13.397123 1553 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:13.411621 2887 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.381769 2843 consensus_queue.cc:1035] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20250114 02:54:13.415251 2887 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.399050 2710 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625)
W20250114 02:54:13.400876 3047 log.cc:926] Time spent T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.107s user 0.000s sys 0.001s
I20250114 02:54:13.400308 2923 raft_consensus.cc:695] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:13.420363 2688 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.421057 2923 consensus_queue.cc:237] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.431471 1553 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:13.434264 1472 catalog_manager.cc:4872] TS b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): tablet 985a08c846764478aaeb97aa5c283bf9 (table test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]) successfully deleted
I20250114 02:54:13.374749 2756 consensus_queue.cc:260] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.444767 1553 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:13.405355 2868 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.448494 1553 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:13.422238 2688 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.422369 2804 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250114 02:54:13.453055 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 79355.1us
I20250114 02:54:13.453960 1702 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:13.450619 3057 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:13.458416 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.455485 1702 leader_election.cc:304] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:13.459829 1654 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:13.449211 2887 raft_consensus.cc:2798] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:13.461153 3057 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:13.463430 1702 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:13.463850 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.463980 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.449801 2878 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.464051 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.470290 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:13.464071 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.451375 2843 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:13.455335 2804 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:13.504357 1662 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:13.509544 1657 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
W20250114 02:54:13.515100 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 51774.2us
I20250114 02:54:13.509934 1656 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:13.517486 1660 raft_consensus.cc:1235] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6. Ops: []
I20250114 02:54:13.464399 3057 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:13.502398 2916 raft_consensus.cc:2743] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:13.466372 2868 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.465035 3065 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:13.463811 2887 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:13.507496 2843 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:13.507576 3009 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.508213 2804 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:13.510541 2841 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.467861 2796 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.538636 3015 raft_consensus.cc:2743] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250114 02:54:13.551129 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 140158us
I20250114 02:54:13.549409 2843 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:13.538349 3061 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.536465 2710 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:13.507144 2878 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.547392 2887 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:13.546648 2916 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:13.574260 1702 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:13.577210 3015 raft_consensus.cc:2743] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:13.578316 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.580801 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.581079 1736 raft_consensus.cc:1270] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:13.582345 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.582823 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
@ 0x7fbcdfa2cfb7 gsignal at ??:0
I20250114 02:54:13.587874 1730 raft_consensus.cc:1235] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:13.590330 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.591012 1730 raft_consensus.cc:2388] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 6: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:13.592952 1736 raft_consensus.cc:1235] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250114 02:54:13.593916 3039 ts_tablet_manager.cc:1918] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 4.2
I20250114 02:54:13.593931 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.594652 1728 raft_consensus.cc:2463] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:13.595765 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.596330 1728 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:13.567721 2756 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:13.569448 3062 raft_consensus.cc:491] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:13.598323 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.606928 1735 raft_consensus.cc:2388] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:13.607656 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.608058 1735 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:13.610256 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.610376 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.610971 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.563280 2841 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.616690 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.617305 1661 raft_consensus.cc:2463] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:13.615100 1733 raft_consensus.cc:2388] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:13.613626 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.619510 1662 raft_consensus.cc:2405] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 02:54:13.615676 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.613112 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.620079 1731 raft_consensus.cc:2388] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:13.621224 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:13.602836 2867 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.626915 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.571790 2688 consensus_queue.cc:1046] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.045s
I20250114 02:54:13.629206 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.604202 2887 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.594477 3039 log.cc:1198] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Deleting WAL directory at /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-1-root/wals/985a08c846764478aaeb97aa5c283bf9
I20250114 02:54:13.630920 1553 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:13.622900 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:13.569552 2916 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.639366 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.639631 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.639986 1585 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:13.640281 1733 raft_consensus.cc:2388] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:13.569696 2710 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.637785 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:13.642467 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.643566 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.644565 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:13.644907 1579 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:13.647753 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:13.602675 2732 consensus_queue.cc:1035] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20250114 02:54:13.651476 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.647948 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:13.652159 1661 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:13.649843 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.652905 1731 raft_consensus.cc:2388] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:13.652859 1587 raft_consensus.cc:2463] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:13.653386 1553 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:13.568322 3061 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.613248 3072 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:13.656409 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.623601 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.657469 1728 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:13.660953 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.655313 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:13.633530 2885 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:13.659734 1584 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:13.660041 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.662494 1586 raft_consensus.cc:1270] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:13.637195 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:13.663993 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:13.651551 2867 raft_consensus.cc:3049] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:13.618244 2843 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.666500 1735 raft_consensus.cc:2388] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:13.671849 2710 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.665716 1733 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:13.686040 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.687171 1581 raft_consensus.cc:2405] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 02:54:13.689074 3083 raft_consensus.cc:2798] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:13.663231 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:13.693707 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:13.700165 1583 raft_consensus.cc:3054] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:13.694234 1579 raft_consensus.cc:2463] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:13.672175 2916 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.699095 1553 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:13.639926 2884 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.664582 3077 raft_consensus.cc:2798] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:13.673040 2867 raft_consensus.cc:738] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:13.617178 3062 raft_consensus.cc:513] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.687470 3068 tablet_bootstrap.cc:492] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd: Bootstrap starting.
I20250114 02:54:13.659015 3063 consensus_queue.cc:1035] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20250114 02:54:13.695459 3083 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:13.725435 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:13.673281 2885 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.712047 2796 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.715698 3077 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:13.732429 1654 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:13.736661 1657 raft_consensus.cc:3054] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 5
I20250114 02:54:13.739277 3082 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.739785 3086 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.740676 3039 ts_tablet_manager.cc:1939] T 985a08c846764478aaeb97aa5c283bf9 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Deleting consensus metadata
I20250114 02:54:13.740345 3086 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.740972 2688 consensus_queue.cc:1046] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:13.741628 3039 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48630 (request call id 59) took 1158 ms. Trace:
I20250114 02:54:13.743031 3074 raft_consensus.cc:491] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:13.743614 3062 leader_election.cc:290] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.743552 3074 raft_consensus.cc:513] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:13.745523 3073 log.cc:926] Time spent T be63bb7ee74c418aabd9237a6f6a0454 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20250114 02:54:13.745965 2867 consensus_queue.cc:260] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.746970 2867 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
W20250114 02:54:13.747114 3071 log.cc:926] Time spent T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.126s user 0.001s sys 0.000s
I20250114 02:54:13.729807 3070 consensus_queue.cc:1035] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.108s
I20250114 02:54:13.749083 2841 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.749590 2841 raft_consensus.cc:3049] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:13.749907 2841 raft_consensus.cc:738] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:13.682602 3072 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.728296 3083 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:13.779629 2887 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.742058 3039 rpcz_store.cc:276] 0114 02:54:12.588311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:12.588872 (+ 561us) service_pool.cc:225] Handling call
0114 02:54:12.660885 (+ 72013us) spinlock_profiling.cc:243] Waited 5.67 ms on lock 0x7b440002494c. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce30ad7c5 00007fbce30ae1e2 00007fbce5d63da1 00007fbce5d5f68a 00007fbce317e81e 00007fbcebd907f9 00007fbce319ba6b 00007fbce3199e6f 00007fbce319bfd2 00007fbceb082794
0114 02:54:12.944323 (+283438us) ts_tablet_manager.cc:1145] Deleting tablet 985a08c846764478aaeb97aa5c283bf9
0114 02:54:12.948720 (+ 4397us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:13.506924 (+558204us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:13.593688 (+ 86764us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:13.741557 (+147869us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36519,"spinlock_wait_cycles":13049600,"tablet-delete.queue_time_us":342734,"thread_start_us":238523,"threads_started":1}
I20250114 02:54:13.729940 3081 raft_consensus.cc:2743] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:13.733371 2878 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.742384 3086 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:13.780946 3076 log.cc:926] Time spent T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250114 02:54:13.734309 3077 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
W20250114 02:54:13.794570 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 103590us
I20250114 02:54:13.796952 1553 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
W20250114 02:54:13.794919 2985 log.cc:926] Time spent T 00000000000000000000000000000000 P e04023df409b4be7b6eb175399d2fd72: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20250114 02:54:13.822625 2911 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.823268 2911 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.825591 2911 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.833781 1657 raft_consensus.cc:2463] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:13.840793 1583 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:13.739799 3082 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.785553 2841 consensus_queue.cc:260] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.861971 2926 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:13.865435 2843 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.910740 1553 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:13.915760 3075 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.916242 3075 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.918246 3075 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:13.920002 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 207825us
I20250114 02:54:13.920218 1702 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:13.905867 2884 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.914996 3061 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.914080 3065 raft_consensus.cc:2743] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:13.927462 1702 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:13.931180 3080 raft_consensus.cc:2743] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:13.931721 3015 consensus_queue.cc:1035] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:13.932271 3015 consensus_queue.cc:1225] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer 1e5870f8269b49cf81dc39f1a5c8f5c1 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 02:54:13.929414 2968 raft_consensus.cc:2798] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:13.933451 2968 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:13.933764 2968 raft_consensus.cc:3054] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:13.934209 1661 raft_consensus.cc:2463] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:13.939783 1702 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:13.955333 1553 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:13.956547 1702 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:13.958618 3082 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.959906 3072 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.921264 2885 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:13.961059 3049 ts_tablet_manager.cc:1918] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 4.2
I20250114 02:54:13.961527 3049 log.cc:1198] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd: Deleting WAL directory at /tmp/dist-test-taskC2NjkR/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1736823160872850-425-0/minicluster-data/ts-0-root/wals/985a08c846764478aaeb97aa5c283bf9
I20250114 02:54:13.880014 3060 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:13.962641 1729 raft_consensus.cc:2463] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:13.969105 3098 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:13.969516 3098 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:13.969882 3098 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
W20250114 02:54:13.980988 2719 consensus_meta.cc:287] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.618s user 0.005s sys 0.003s
I20250114 02:54:13.982774 3060 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:13.962684 3075 raft_consensus.cc:2743] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:13.938899 3062 consensus_queue.cc:1035] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
W20250114 02:54:13.987913 3088 log.cc:926] Time spent T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20250114 02:54:13.991236 3092 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:13.994822 3065 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:13.995326 3065 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:13.995430 3049 ts_tablet_manager.cc:1939] T 985a08c846764478aaeb97aa5c283bf9 P 62daa58e8ff64698bb46ea3fd5290cdd: Deleting consensus metadata
I20250114 02:54:13.995771 3065 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.005143 2804 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:14.020059 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 54454.4us
I20250114 02:54:14.021296 1702 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:14.023432 3049 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34912 (request call id 59) took 1141 ms. Trace:
W20250114 02:54:14.026046 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 71014us
I20250114 02:54:14.026990 1553 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.029031 3074 leader_election.cc:290] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:14.032590 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 63698.4us
I20250114 02:54:14.036538 3074 raft_consensus.cc:2743] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:14.037458 1553 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.038480 1553 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.039474 1553 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.040251 3086 raft_consensus.cc:2743] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:14.040127 3083 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.041067 3086 raft_consensus.cc:2743] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:14.041671 3086 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
W20250114 02:54:14.044420 3013 consensus_meta.cc:287] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.661s user 0.006s sys 0.003s
I20250114 02:54:14.044780 3013 raft_consensus.cc:2743] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:14.045693 1731 raft_consensus.cc:1235] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 3. Current term is 4. Ops: []
W20250114 02:54:14.046739 3057 consensus_meta.cc:287] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.505s user 0.009s sys 0.000s
I20250114 02:54:14.047107 3057 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.047983 3075 raft_consensus.cc:2798] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:14.048341 3075 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.048645 3075 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.055570 1553 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.042325 3083 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.059067 1628 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:14.060056 1717 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36426 (request call id 60) took 1196 ms. Trace:
I20250114 02:54:14.060817 1702 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:14.063190 1628 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:14.064620 1628 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:14.065877 1702 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:14.066658 3089 raft_consensus.cc:2798] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:14.067056 3089 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.067378 3089 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.060456 3097 raft_consensus.cc:2743] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:14.070344 2841 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:14.072181 3062 raft_consensus.cc:2798] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:14.072696 3062 raft_consensus.cc:695] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:14.073489 3057 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.062674 2885 raft_consensus.cc:2758] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250114 02:54:14.056211 3074 consensus_queue.cc:1035] T 328b539725d54455aadcb0e76e17b4d6 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20250114 02:54:14.074895 1553 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.061846 2926 raft_consensus.cc:2743] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:14.023907 3049 rpcz_store.cc:276] 0114 02:54:12.882624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:12.886756 (+ 4132us) service_pool.cc:225] Handling call
0114 02:54:12.947987 (+ 61231us) ts_tablet_manager.cc:1145] Deleting tablet 985a08c846764478aaeb97aa5c283bf9
0114 02:54:12.948063 (+ 76us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:13.654466 (+706403us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:13.960808 (+306342us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:14.023308 (+ 62500us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98923,"tablet-delete.queue_time_us":58041,"thread_start_us":30512,"threads_started":1}
I20250114 02:54:14.079120 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.079301 2868 raft_consensus.cc:2798] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:14.065878 2867 raft_consensus.cc:2798] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:14.079346 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.077081 2804 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.080224 3074 raft_consensus.cc:2743] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:14.080677 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:14.093014 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.093647 1582 raft_consensus.cc:2388] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:14.094102 1583 raft_consensus.cc:2405] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:14.081637 2868 raft_consensus.cc:695] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
@ 0x7fbcdfa2e921 abort at ??:0
I20250114 02:54:14.095301 2868 consensus_queue.cc:237] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
@ 0x2cf166 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
I20250114 02:54:14.098460 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.085379 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.099668 1661 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:14.088485 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.082024 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.102365 1580 raft_consensus.cc:2388] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 5: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:14.098793 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.119719 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4d46461bf745639ea4f64fa3bcb996" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:14.084226 3099 log.cc:926] Time spent T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.155s user 0.002s sys 0.000s
I20250114 02:54:14.126982 1553 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:14.127378 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.127537 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.129086 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.128762 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:14.128163 1583 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:14.128211 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:14.133776 1733 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:14.134321 1553 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.134661 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:14.135512 1736 raft_consensus.cc:1270] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:54:14.126909 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:14.131232 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:14.137380 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.138121 1656 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:14.138255 1702 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:14.086812 2867 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.139568 1702 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:14.129657 1582 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:14.142068 1553 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:14.142647 1702 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:14.143040 1736 raft_consensus.cc:1270] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:54:14.143265 1553 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:14.143823 1702 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:14.145283 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.145593 1734 raft_consensus.cc:1235] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 02:54:14.062841 3098 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.146068 1658 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:14.060503 1717 rpcz_store.cc:276] 0114 02:54:12.863484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:12.863623 (+ 139us) service_pool.cc:225] Handling call
0114 02:54:12.874138 (+ 10515us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0114 02:54:12.874260 (+ 122us) ts_tablet_manager.cc:676] Creating new metadata...
0114 02:54:13.536087 (+661827us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:13.961145 (+425058us) tablet_replica.cc:206] Creating consensus instance
0114 02:54:14.059729 (+ 98584us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25833856,"tablet-open.queue_time_us":101294,"thread_start_us":74207,"threads_started":1}
I20250114 02:54:14.146548 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:14.136776 3068 tablet_bootstrap.cc:700] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent opening tablet: real 0.104s user 0.002s sys 0.000s
I20250114 02:54:14.148907 1581 raft_consensus.cc:1235] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250114 02:54:14.148080 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:14.150625 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.150902 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.159619 1658 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 6.
I20250114 02:54:14.150933 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.152594 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:14.089200 3057 consensus_queue.cc:1035] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.052s
I20250114 02:54:14.126884 3061 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:14.181448 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.141253 2796 raft_consensus.cc:2743] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:14.153609 1628 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.140901 2867 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:14.183615 1702 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:14.149307 3102 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.161607 1587 raft_consensus.cc:2363] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5.
I20250114 02:54:14.191187 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.193325 1628 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.123951 1472 catalog_manager.cc:4872] TS 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): tablet 985a08c846764478aaeb97aa5c283bf9 (table test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]) successfully deleted
I20250114 02:54:14.147917 3060 raft_consensus.cc:2743] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:14.196947 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:54:14.198786 1472 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 74857.6us
I20250114 02:54:14.199414 1579 raft_consensus.cc:2405] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 02:54:14.152786 2732 raft_consensus.cc:2743] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:14.116439 3065 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.204311 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.204895 1583 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:14.163247 1585 raft_consensus.cc:1270] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250114 02:54:14.140894 2719 raft_consensus.cc:2743] T 0ce7223c99df448483de68f3463ba115 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:14.151794 1661 pending_rounds.cc:77] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Aborting all ops after (but not including) 1
I20250114 02:54:14.151225 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.073527 3062 consensus_queue.cc:237] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.140964 2804 raft_consensus.cc:2798] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:14.149101 3015 consensus_queue.cc:1035] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20250114 02:54:14.234998 1731 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
W20250114 02:54:14.236114 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 52600.5us
I20250114 02:54:14.148082 3072 raft_consensus.cc:2798] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:14.212280 1661 pending_rounds.cc:99] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 02:54:14.245522 1661 raft_consensus.cc:2883] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:14.149226 3080 raft_consensus.cc:2743] T 8c49b6c0ab1d49079166b6a4cdf59896 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:14.240036 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:14.246412 1736 raft_consensus.cc:2388] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:14.240609 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:14.148135 2843 raft_consensus.cc:2798] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:14.208235 3068 tablet_bootstrap.cc:654] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.244273 1702 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:14.247567 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:14.242241 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:14.255107 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.198809 3063 raft_consensus.cc:2743] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:14.270659 1587 raft_consensus.cc:1235] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: []
W20250114 02:54:14.270689 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 67214us
I20250114 02:54:14.272011 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:14.272755 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c49b6c0ab1d49079166b6a4cdf59896" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:14.269961 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:14.293967 1736 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.294719 1553 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:14.207991 2968 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.277755 1628 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:14.278578 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.208004 2923 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.278488 1655 raft_consensus.cc:2388] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:14.279265 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:14.269346 3083 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:14.300539 1730 raft_consensus.cc:2388] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:14.271560 1659 raft_consensus.cc:1235] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6. Ops: []
I20250114 02:54:14.283744 3103 consensus_queue.cc:1035] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20250114 02:54:14.216207 3102 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.194765 3061 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.234853 2804 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.251910 2843 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:14.281986 1646 tablet_service.cc:1467] Processing CreateTablet for tablet c4ff92fc16244b87b6d30cf38194182d (DEFAULT_TABLE table=test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250114 02:54:14.289280 1658 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:14.245034 3072 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.248872 3060 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:14.252967 1702 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:14.192924 3077 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.279633 2916 raft_consensus.cc:2743] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:14.316223 1553 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:14.318506 1586 raft_consensus.cc:1235] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250114 02:54:14.326762 1727 raft_consensus.cc:2388] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 4: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:14.331681 1553 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:14.332726 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.303223 3057 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:14.335404 1733 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
W20250114 02:54:14.337910 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 84993.7us
I20250114 02:54:14.313869 2804 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:14.317031 2843 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:14.337172 3060 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.314014 3074 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.354033 1702 leader_election.cc:304] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:14.375533 1702 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:14.334995 3015 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.335381 3081 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:14.327106 3072 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.347707 3009 raft_consensus.cc:2798] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:14.302651 2923 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:14.412698 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 75254.9us
I20250114 02:54:14.304917 3083 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.344489 3057 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:14.381603 3090 raft_consensus.cc:2743] T 0ecc21b2d4ec4b0aab2d25073324044b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:14.392462 3015 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.398613 1646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4ff92fc16244b87b6d30cf38194182d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.373827 3060 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.425644 1660 raft_consensus.cc:2463] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:14.402069 1656 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:14.330864 2887 consensus_queue.cc:1046] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20250114 02:54:14.354926 3097 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:14.382516 3062 raft_consensus.cc:2743] T 8d4d46461bf745639ea4f64fa3bcb996 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:14.374773 3098 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:14.413455 3057 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:14.401504 2710 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:14.410980 3009 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:14.408632 3086 raft_consensus.cc:2743] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:14.435878 3009 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.414378 3065 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:14.436578 3097 raft_consensus.cc:695] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:14.437634 3097 consensus_queue.cc:237] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.407569 3081 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:14.438747 3081 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.421989 2923 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.380156 3074 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.419381 3083 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.435418 3102 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:14.442150 1580 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:14.410651 3092 raft_consensus.cc:2743] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:14.454262 3060 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:14.455001 3112 log.cc:926] Time spent T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20250114 02:54:14.209190 1472 catalog_manager.cc:4872] TS 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625): tablet 985a08c846764478aaeb97aa5c283bf9 (table test-workload [id=58ecc04d502c4a16a29a47fd3054e79e]) successfully deleted
W20250114 02:54:14.460928 1472 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 251754us
W20250114 02:54:14.461561 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20250114 02:54:14.462668 3109 log.cc:926] Time spent T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
W20250114 02:54:14.467293 3118 log.cc:926] Time spent T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20250114 02:54:14.433806 2710 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.477262 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:14.482069 3102 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:14.482501 3102 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.482898 3061 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:14.484306 3102 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:14.484982 3089 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.485427 2843 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.486382 1657 raft_consensus.cc:2388] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 5: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:14.486487 1581 raft_consensus.cc:1235] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:14.486729 3077 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.473209 2968 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.482450 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:14.487882 2710 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:14.488339 3089 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.482957 3062 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:14.494995 3062 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.495990 1736 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:14.496665 3062 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:14.496487 3075 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.497890 3015 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:14.490084 2867 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.500618 2804 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.501812 3074 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:14.504891 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 91661.1us
W20250114 02:54:14.506202 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 228529us
I20250114 02:54:14.515955 1553 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:14.516017 1628 leader_election.cc:304] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.502188 3072 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.475548 2997 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:14.525492 1553 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:14.513147 2756 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:14.512969 2913 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.528849 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.530005 1662 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:14.517639 2843 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.531327 1553 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:14.526877 2968 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:14.578157 1662 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:14.587255 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.601382 1702 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:14.586959 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:14.592769 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 62301us
I20250114 02:54:14.529559 2719 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:14.564870 1653 raft_consensus.cc:1270] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:14.553967 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.617323 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.619264 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.529629 2756 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:14.591604 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.622215 1628 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.659508 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:54:14.660593 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 59332us
I20250114 02:54:14.660627 1580 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:14.661661 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.662243 1657 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.610383 1660 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:14.663537 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:14.664052 1586 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.659693 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.566234 2971 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.569591 3106 tablet_bootstrap.cc:492] T c4ff92fc16244b87b6d30cf38194182d P b89550620e1d455d887be72232bb5454: Bootstrap starting.
I20250114 02:54:14.640722 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:14.665316 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 134061us
W20250114 02:54:14.667596 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 45506.5us
I20250114 02:54:14.669051 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.671375 1655 raft_consensus.cc:1270] T 4876e2bdbafa4d8684adc1348f1aeb80 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:14.671943 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.672525 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:14.673004 3068 tablet_bootstrap.cc:492] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd: No bootstrap required, opened a new log
I20250114 02:54:14.673271 1483 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39724 (request call id 113) took 4578 ms. Trace:
I20250114 02:54:14.667061 2719 raft_consensus.cc:695] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:14.679021 3103 raft_consensus.cc:2743] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 02:54:14.558938 3075 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:14.665895 2971 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.567508 3072 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.547823 2867 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.680737 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.681378 3077 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:14.682843 3062 raft_consensus.cc:2798] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:14.684279 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.528875 2997 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.679431 2878 raft_consensus.cc:491] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
W20250114 02:54:14.662307 3126 log.cc:926] Time spent T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.073s user 0.002s sys 0.006s
I20250114 02:54:14.532949 3070 raft_consensus.cc:2743] T daf552d03bce48dda4e5b9772287649a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:14.731114 1653 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.580075 3060 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:14.575338 3124 raft_consensus.cc:3049] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Stepping down as leader of term 5
I20250114 02:54:14.688453 1702 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:14.736274 1660 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.549165 2804 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.740381 1657 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:14.611326 2968 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.742624 1628 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:14.535030 3083 consensus_queue.cc:1046] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.201s
I20250114 02:54:14.686998 2756 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:14.744488 1656 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:14.747076 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.749699 3062 raft_consensus.cc:695] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:14.750273 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.750957 2719 consensus_queue.cc:237] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.751437 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.770449 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.569738 3009 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.533399 2913 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.545472 3057 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:14.786509 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 43997.7us
I20250114 02:54:14.779953 1553 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.781854 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.787536 1657 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.787986 1628 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:14.751389 2710 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.538081 3081 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.785365 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.790730 1654 raft_consensus.cc:2388] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:14.777482 3068 ts_tablet_manager.cc:1397] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent bootstrapping tablet: real 1.144s user 0.034s sys 0.027s
I20250114 02:54:14.791819 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:14.754264 2867 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.803480 2841 consensus_queue.cc:1035] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:54:14.803973 2867 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.805794 2867 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.754293 3124 raft_consensus.cc:738] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:14.807226 3124 consensus_queue.cc:260] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.808354 3124 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
W20250114 02:54:14.799206 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 110845us
I20250114 02:54:14.799346 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.810551 1702 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:14.801780 1553 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.753366 3129 consensus_queue.cc:1035] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20250114 02:54:14.810637 1658 raft_consensus.cc:2463] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 2.
I20250114 02:54:14.811769 1702 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:14.728935 1483 rpcz_store.cc:276] 0114 02:54:10.096989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:10.101247 (+ 4258us) service_pool.cc:225] Handling call
0114 02:54:14.673199 (+4571952us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:12.742700 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:12.783070 (+ 40370us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:12.783118 (+ 48us) write_op.cc:435] Acquired schema lock
0114 02:54:12.783143 (+ 25us) tablet.cc:659] Decoding operations
0114 02:54:12.783883 (+ 740us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:12.783947 (+ 64us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:12.783975 (+ 28us) tablet.cc:682] Acquiring locks for 5 operations
0114 02:54:12.784606 (+ 631us) tablet.cc:698] Row locks acquired
0114 02:54:12.784626 (+ 20us) write_op.cc:260] PREPARE: finished
0114 02:54:12.784757 (+ 131us) write_op.cc:270] Start()
0114 02:54:12.784878 (+ 121us) write_op.cc:276] Timestamp: P: 1736823252784713 usec, L: 0
0114 02:54:12.784912 (+ 34us) op_driver.cc:348] REPLICATION: starting
0114 02:54:12.785509 (+ 597us) log.cc:844] Serialized 2078 byte log entry
0114 02:54:13.383864 (+598355us) op_driver.cc:464] REPLICATION: finished
0114 02:54:13.455539 (+ 71675us) write_op.cc:301] APPLY: starting
0114 02:54:13.460958 (+ 5419us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:13.509835 (+ 48877us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:13.540130 (+ 30295us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:13.663685 (+123555us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:13.710968 (+ 47283us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0114 02:54:13.714257 (+ 3289us) write_op.cc:312] APPLY: finished
0114 02:54:13.730117 (+ 15860us) log.cc:844] Serialized 46 byte log entry
0114 02:54:13.736335 (+ 6218us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:13.751832 (+ 15497us) write_op.cc:454] Released schema lock
0114 02:54:13.782005 (+ 30173us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:13.813142 (+ 31137us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:13.818617 (+ 5475us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":329003,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":53067,"apply.run_cpu_time_us":3648,"apply.run_wall_time_us":346358,"num_ops":5,"prepare.queue_time_us":529,"prepare.run_cpu_time_us":3371,"prepare.run_wall_time_us":51985,"replication_time_us":577130,"thread_start_us":33008,"threads_started":1}]]}
I20250114 02:54:14.817045 1702 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:14.785980 3090 raft_consensus.cc:2798] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:14.812278 1657 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:14.791285 1659 raft_consensus.cc:2388] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:14.820160 1553 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.797709 1660 raft_consensus.cc:2463] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:14.820993 1702 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:14.741804 3077 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.792372 3063 raft_consensus.cc:2798] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:14.789736 2968 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:14.828779 1553 leader_election.cc:304] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.744899 2878 raft_consensus.cc:513] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.831017 1553 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:14.819171 3083 raft_consensus.cc:2743] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 02:54:14.833189 1702 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:14.825193 3060 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.818782 2796 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:14.822012 1578 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.857043 1702 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:14.833612 1628 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:14.860625 1581 raft_consensus.cc:1235] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:14.867662 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:14.868330 1733 raft_consensus.cc:3054] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:14.874424 1736 raft_consensus.cc:1235] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250114 02:54:14.874713 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:14.877038 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:14.877633 1735 raft_consensus.cc:2463] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:54:14.875751 1579 raft_consensus.cc:2463] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 2.
I20250114 02:54:14.831918 3090 raft_consensus.cc:695] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:14.832192 3063 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.828567 3065 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:14.884364 3114 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:14.887933 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:14.888248 2878 leader_election.cc:290] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:14.887816 3090 consensus_queue.cc:237] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.889021 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:14.833016 3074 raft_consensus.cc:2743] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:14.837693 2868 raft_consensus.cc:2798] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:14.837371 3123 raft_consensus.cc:2743] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:14.892164 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:14.833526 3097 raft_consensus.cc:2743] T 8c49b6c0ab1d49079166b6a4cdf59896 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:14.855197 2887 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:14.832703 3082 consensus_queue.cc:1035] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20250114 02:54:14.912181 1583 raft_consensus.cc:2388] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 6: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:14.888626 3117 raft_consensus.cc:2798] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:14.918501 1580 raft_consensus.cc:3054] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.884099 2732 raft_consensus.cc:2798] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:14.889569 3115 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.894397 3063 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
W20250114 02:54:14.925194 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 45018.1us
I20250114 02:54:14.901091 3114 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:14.930274 1702 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:14.918967 2868 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.917156 3065 raft_consensus.cc:695] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:14.931902 1553 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:14.921455 3103 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:14.920794 3116 consensus_queue.cc:1035] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.093s
I20250114 02:54:14.915467 2796 raft_consensus.cc:695] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:14.936174 1628 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.922546 3117 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.952966 1553 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:14.944976 2710 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:14.953897 3137 tablet_bootstrap.cc:492] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Bootstrap starting.
I20250114 02:54:14.955268 1628 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:14.931378 2867 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.959956 1580 raft_consensus.cc:2463] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:14.893253 3062 consensus_queue.cc:237] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.939296 3114 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:14.943989 2868 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:14.938881 2732 raft_consensus.cc:695] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:14.945052 3122 raft_consensus.cc:3049] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Stepping down as leader of term 5
I20250114 02:54:14.944886 2971 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.981328 1660 raft_consensus.cc:1235] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6. Ops: []
I20250114 02:54:14.948755 2887 raft_consensus.cc:695] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:14.945772 3103 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.954397 3097 raft_consensus.cc:2798] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:54:14.954293 3140 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.954551 2997 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.988598 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:14.960445 3129 raft_consensus.cc:2798] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election won for term 5
W20250114 02:54:15.075752 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 83821us
I20250114 02:54:14.956338 3117 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:14.960978 3098 raft_consensus.cc:2743] T c1f2bc8299cf4834bebd9aa5ae95a56e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:14.965992 3057 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:14.931563 3115 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:14.975881 3009 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:14.986001 3082 raft_consensus.cc:2743] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 02:54:14.988704 3103 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:14.984997 3122 raft_consensus.cc:738] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:15.099139 1586 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
W20250114 02:54:15.101258 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 143079us
I20250114 02:54:15.103494 1653 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:14.991634 3097 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:15.079046 3123 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.106074 1628 leader_election.cc:400] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5.
I20250114 02:54:15.107525 1628 leader_election.cc:403] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250114 02:54:15.087814 2913 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:15.086484 3129 raft_consensus.cc:695] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:14.970804 2867 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.095935 3077 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:15.096138 3141 raft_consensus.cc:2743] T a26544b19eeb4116a06caaba1da0e9d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 02:54:15.096530 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:15.099860 2843 raft_consensus.cc:2758] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20250114 02:54:15.111129 3145 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.086386 2923 consensus_queue.cc:1046] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20250114 02:54:15.107188 3072 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.106799 3097 raft_consensus.cc:3054] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:15.106729 3009 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:14.990707 3089 consensus_queue.cc:1046] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.114s
I20250114 02:54:15.109900 2971 raft_consensus.cc:2743] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5.
I20250114 02:54:15.125155 3139 consensus_queue.cc:1035] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
W20250114 02:54:15.128412 3138 log.cc:926] Time spent T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.180s user 0.000s sys 0.001s
I20250114 02:54:15.095528 2796 consensus_queue.cc:237] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.143512 3129 consensus_queue.cc:237] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.079337 3140 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.158679 1733 raft_consensus.cc:2463] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:15.110229 3123 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.168973 3152 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:15.125533 2887 consensus_queue.cc:237] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.117964 2732 consensus_queue.cc:237] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.130134 3072 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.187605 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:15.253157 1657 raft_consensus.cc:2363] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
W20250114 02:54:15.195246 3142 log.cc:926] Time spent T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.230s user 0.000s sys 0.001s
I20250114 02:54:15.196408 3083 raft_consensus.cc:3049] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Stepping down as leader of term 5
I20250114 02:54:15.104266 3065 consensus_queue.cc:237] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.147238 2867 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:15.174027 2756 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.151721 3116 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:15.130782 3009 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.166218 3150 raft_consensus.cc:3049] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 02:54:15.263000 3116 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.270495 1578 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:15.270793 3116 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:15.271235 1656 raft_consensus.cc:1235] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6. Ops: []
I20250114 02:54:15.135689 3154 consensus_queue.cc:1046] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.124s
W20250114 02:54:15.274906 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 335697us
I20250114 02:54:15.275321 3154 raft_consensus.cc:3049] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:15.275821 3154 raft_consensus.cc:738] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:15.277784 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:15.278950 1586 raft_consensus.cc:1270] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:15.281049 1579 raft_consensus.cc:1270] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:15.280990 2878 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:15.281585 2878 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.282045 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:15.282922 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:15.282985 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:15.283439 2878 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:15.283813 1585 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:15.284138 2878 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.284588 2878 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.286360 2878 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:15.283833 1581 raft_consensus.cc:2388] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 5: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:15.287384 2878 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:15.287817 2878 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.288018 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:15.288185 3125 raft_consensus.cc:491] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:15.288655 3125 raft_consensus.cc:513] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.289038 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:15.289741 1583 raft_consensus.cc:2388] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:15.290181 2916 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:15.283540 1658 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:15.290807 2916 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.292568 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:15.293236 1662 raft_consensus.cc:1270] T b4aa8b25b08141db9f0576ce3fa34d3c P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:15.294314 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:15.295138 3075 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:15.295687 3075 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.292984 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:15.293110 1734 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:15.300951 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e023d568848b9861c2976a138d461" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:15.276777 3154 consensus_queue.cc:260] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.308779 3154 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:15.312074 3125 leader_election.cc:290] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:15.312695 2878 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:15.312973 2916 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:15.313694 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:15.312650 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:15.316351 1734 raft_consensus.cc:3054] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:15.316510 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:15.317276 1729 raft_consensus.cc:2463] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 1.
I20250114 02:54:15.318311 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:15.320729 3075 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:15.320994 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:15.321780 1729 raft_consensus.cc:2388] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:15.322978 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:15.323570 1729 raft_consensus.cc:3054] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:15.321025 1658 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:15.336283 1729 raft_consensus.cc:2463] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:15.336880 1585 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:15.332511 1702 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:15.340027 1702 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:15.340907 2916 raft_consensus.cc:2798] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:15.341476 2916 raft_consensus.cc:695] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:15.341645 1702 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:15.337769 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:15.342247 2916 consensus_queue.cc:237] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.342875 1729 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:15.343725 3013 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:15.344027 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:15.344218 3013 raft_consensus.cc:695] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:15.344969 1729 raft_consensus.cc:2388] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:15.345213 3013 consensus_queue.cc:237] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.348414 2878 consensus_queue.cc:1035] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:54:15.349704 1702 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:15.351943 1730 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:15.151666 3068 raft_consensus.cc:357] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.257467 3083 raft_consensus.cc:738] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:15.128976 3145 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.345842 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:15.264869 3150 raft_consensus.cc:738] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:15.360030 1734 raft_consensus.cc:2405] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250114 02:54:15.360620 1553 leader_election.cc:400] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
I20250114 02:54:15.376852 1553 leader_election.cc:403] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250114 02:54:15.212910 3152 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.398166 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:15.398784 2884 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.399646 3152 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:15.399310 2884 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.398900 1655 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:15.400940 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:15.423123 1653 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:15.401787 1657 raft_consensus.cc:1235] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20250114 02:54:15.411092 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 50652.7us
W20250114 02:54:15.376029 3142 log.cc:926] Time spent T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20250114 02:54:15.140424 3122 consensus_queue.cc:260] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.357295 3068 raft_consensus.cc:383] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.393185 3125 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:15.420197 1702 leader_election.cc:304] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:15.421905 3086 raft_consensus.cc:2743] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5.
W20250114 02:54:15.446206 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 306807us
I20250114 02:54:15.421320 3115 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:15.423008 3072 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:15.469930 1553 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:15.429144 3068 raft_consensus.cc:738] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.442898 3081 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:15.476416 1729 raft_consensus.cc:1235] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6. Ops: []
I20250114 02:54:15.477214 1734 raft_consensus.cc:1270] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:15.472111 2756 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:15.626286 3137 tablet_bootstrap.cc:700] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent opening tablet: real 0.144s user 0.000s sys 0.005s
I20250114 02:54:15.627926 1628 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:15.451093 3125 raft_consensus.cc:695] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:15.438268 3083 consensus_queue.cc:260] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:15.636991 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 167137us
I20250114 02:54:15.471315 3155 raft_consensus.cc:2743] T 3a7e023d568848b9861c2976a138d461 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:15.504918 3116 consensus_queue.cc:1035] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.173s
I20250114 02:54:15.503845 2867 consensus_queue.cc:1046] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.173s
I20250114 02:54:15.476913 2796 consensus_queue.cc:1035] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.162s
I20250114 02:54:15.641914 3074 raft_consensus.cc:2798] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:15.638881 2885 consensus_queue.cc:1046] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
I20250114 02:54:15.695075 2878 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:15.450387 3150 consensus_queue.cc:260] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
@ 0x7fbce2825da4 google::LogMessage::SendToLog() at ??:0
I20250114 02:54:15.796032 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 636) took 1049 ms. Trace:
W20250114 02:54:15.705893 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 229671us
I20250114 02:54:15.815806 1628 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
W20250114 02:54:15.824918 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:54:15.629244 3068 consensus_queue.cc:260] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.821120 1656 rpcz_store.cc:276] 0114 02:54:14.746744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.814850 (+ 68106us) service_pool.cc:225] Handling call
0114 02:54:14.827944 (+ 13094us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:15.274879 (+446935us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:15.795815 (+520936us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b5400124dd4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:15.795976 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1019166592}
I20250114 02:54:15.828799 1702 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:15.834906 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:15.835139 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:15.836148 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:15.837455 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:15.838011 1580 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:15.836726 1579 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:15.847116 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:15.847991 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:15.848651 1580 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:15.849676 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:15.861749 1553 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
W20250114 02:54:15.783478 3161 log.cc:926] Time spent T 328b539725d54455aadcb0e76e17b4d6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.154s user 0.000s sys 0.000s
I20250114 02:54:15.723075 3137 tablet_bootstrap.cc:654] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.863647 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:15.921075 1579 raft_consensus.cc:2388] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 6: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
W20250114 02:54:15.782924 3156 log.cc:926] Time spent T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.197s user 0.001s sys 0.000s
I20250114 02:54:15.879321 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:15.880873 2710 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:15.705093 3083 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:15.719313 2884 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:15.756464 2756 raft_consensus.cc:491] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:15.882261 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:15.817005 3150 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:15.883296 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:15.936313 1657 raft_consensus.cc:2405] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250114 02:54:15.936465 1580 raft_consensus.cc:2388] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:15.883565 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:15.935292 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:15.789904 3013 raft_consensus.cc:491] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:15.887904 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:54:15.890786 3133 log.cc:926] Time spent T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20250114 02:54:15.900108 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:15.927142 3160 raft_consensus.cc:2798] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:15.902837 3068 ts_tablet_manager.cc:1428] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent starting tablet: real 1.093s user 0.001s sys 0.010s
W20250114 02:54:15.905740 3144 log.cc:926] Time spent T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.920s user 0.004s sys 0.002s
I20250114 02:54:15.905974 2843 raft_consensus.cc:2798] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:15.906755 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:54:15.890496 3063 consensus_meta.cc:287] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.951s user 0.005s sys 0.001s
I20250114 02:54:15.908617 3077 raft_consensus.cc:491] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:15.909255 3072 raft_consensus.cc:491] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:15.909435 1578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 571) took 1433 ms. Trace:
I20250114 02:54:15.751895 2916 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.723186 3140 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:15.724356 3074 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:15.756302 3081 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:15.934618 2710 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.721055 3075 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:15.744333 3155 raft_consensus.cc:491] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.750677 3123 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:15.962411 3160 raft_consensus.cc:695] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:15.911422 1553 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
W20250114 02:54:15.915351 2868 consensus_meta.cc:287] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.922s user 0.006s sys 0.003s
I20250114 02:54:15.719568 3158 consensus_queue.cc:1046] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.445s
I20250114 02:54:15.777962 2878 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.794742 3129 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:15.749840 2923 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.783658 3131 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
W20250114 02:54:15.719318 3127 log.cc:926] Time spent T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250114 02:54:15.921146 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:15.782435 3157 log.cc:926] Time spent T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.196s user 0.000s sys 0.000s
W20250114 02:54:15.701895 3109 log.cc:926] Time spent T 0b78459d23a744819d55e5f50de8174f P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
W20250114 02:54:15.807541 3163 log.cc:926] Time spent T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.130s user 0.000s sys 0.000s
W20250114 02:54:15.808123 3162 log.cc:926] Time spent T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20250114 02:54:15.881636 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce7223c99df448483de68f3463ba115" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:15.929934 1653 raft_consensus.cc:2388] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 5: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:15.931589 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:15.883291 3106 tablet_bootstrap.cc:654] T c4ff92fc16244b87b6d30cf38194182d P b89550620e1d455d887be72232bb5454: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.707002 3009 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:15.808874 3145 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:15.886297 1730 consensus_meta.cc:287] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.534s user 0.005s sys 0.000s
I20250114 02:54:16.033133 1730 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:15.854939 3102 raft_consensus.cc:2743] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250114 02:54:15.928073 3090 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.019s sys 0.033s
W20250114 02:54:16.053566 3090 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s user 0.020s sys 0.033s
W20250114 02:54:15.901400 3117 consensus_meta.cc:287] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.807s user 0.004s sys 0.004s
I20250114 02:54:16.054378 3117 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:16.055146 3097 consensus_meta.cc:287] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 0.924s user 0.009s sys 0.000s
I20250114 02:54:16.055492 3097 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.056463 3117 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:15.955473 3149 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.057448 3149 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.059993 3149 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.061513 1732 raft_consensus.cc:2388] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:15.956027 1628 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:15.944958 1581 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:16.062667 1736 raft_consensus.cc:1235] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 2. Current term is 3. Ops: []
I20250114 02:54:16.062973 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:15.960799 1584 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:15.907755 3057 raft_consensus.cc:491] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.770350 3125 consensus_queue.cc:237] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.937963 3092 raft_consensus.cc:491] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.956789 3013 raft_consensus.cc:513] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.958582 1655 raft_consensus.cc:2388] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:15.965193 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:15.962872 3060 consensus_meta.cc:287] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 1.085s user 0.010s sys 0.001s
W20250114 02:54:15.960647 2968 consensus_meta.cc:287] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd: Time spent flushing consensus metadata: real 1.126s user 0.009s sys 0.002s
I20250114 02:54:15.970927 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:15.960512 2843 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:15.719470 3153 raft_consensus.cc:2798] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250114 02:54:15.961717 3068 ts_tablet_manager.cc:1460] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd: Tablet startup took 1162ms
I20250114 02:54:15.975020 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:15.942081 2756 raft_consensus.cc:513] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.750041 3062 raft_consensus.cc:491] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
W20250114 02:54:15.981766 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 70462.9us
I20250114 02:54:15.973331 3074 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:15.993019 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 749) took 1119 ms. Trace:
I20250114 02:54:15.964740 3063 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.968312 3072 raft_consensus.cc:513] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.011307 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:15.965610 3077 raft_consensus.cc:513] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.988821 3123 raft_consensus.cc:2798] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:15.989033 3152 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:16.015908 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.016404 3178 raft_consensus.cc:3049] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 02:54:16.017068 2710 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.017462 1702 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:15.984264 2868 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.020659 3116 raft_consensus.cc:491] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.977221 3081 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.023955 2878 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.007504 3131 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.007755 2923 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.966243 1578 rpcz_store.cc:276] 0114 02:54:14.475636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.482209 (+ 6573us) service_pool.cc:225] Handling call
0114 02:54:15.909245 (+1427036us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b54001ab8d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:15.909347 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":753963776}
W20250114 02:54:16.004280 3103 consensus_meta.cc:287] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent flushing consensus metadata: real 0.900s user 0.010s sys 0.000s
I20250114 02:54:15.971694 2916 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.033279 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.092044 3145 raft_consensus.cc:2798] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:16.002900 3129 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:15.979233 3155 raft_consensus.cc:513] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.092191 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.180431 1733 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:16.179199 1583 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
W20250114 02:54:16.178032 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 160662us
I20250114 02:54:16.182158 1702 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:16.092732 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.178957 1587 rpcz_store.cc:276] 0114 02:54:14.865530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.865639 (+ 109us) service_pool.cc:225] Handling call
0114 02:54:14.888535 (+ 22896us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:14.927333 (+ 38798us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:14.927361 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:14.927399 (+ 38us) raft_consensus.cc:1640] Marking committed up to 3
0114 02:54:14.927424 (+ 25us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:15.984376 (+1056952us) spinlock_profiling.cc:243] Waited 38.6 ms on lock 0x7b54001ab3d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:15.984416 (+ 40us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:15.984541 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":88827264}
I20250114 02:54:16.179749 1657 raft_consensus.cc:2463] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:16.093418 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:16.185297 1653 raft_consensus.cc:2363] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6.
I20250114 02:54:16.187359 1702 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:16.187633 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.191167 1702 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:16.188486 1733 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:16.193128 1729 raft_consensus.cc:2388] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 6: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:16.186123 2710 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.193750 3102 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:16.193974 2710 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.194304 3102 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:16.195767 3114 consensus_meta.cc:287] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 1.204s user 0.000s sys 0.010s
I20250114 02:54:16.195917 2710 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.196012 3102 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.196475 2710 raft_consensus.cc:2743] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:16.196631 3117 raft_consensus.cc:2798] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:16.196985 3117 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:16.197316 3117 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:16.197572 1586 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:16.198137 2878 raft_consensus.cc:2798] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:16.200214 3155 leader_election.cc:290] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.183128 3060 raft_consensus.cc:2743] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:16.203847 3063 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:16.204704 3159 log.cc:926] Time spent T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.527s user 0.002s sys 0.000s
I20250114 02:54:16.205660 1658 raft_consensus.cc:1270] T 30486d50dbd94c78a14bad5b98494276 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:16.207461 3081 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.207801 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.208375 1578 raft_consensus.cc:3054] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:16.178457 3178 raft_consensus.cc:738] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:16.211958 1660 raft_consensus.cc:2363] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 5. Current term is 6.
I20250114 02:54:16.219831 3074 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:16.220662 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 264735us
I20250114 02:54:16.217268 3122 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:16.212502 1731 raft_consensus.cc:2463] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:16.179996 3153 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.234854 1728 raft_consensus.cc:1235] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250114 02:54:16.183828 2843 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
W20250114 02:54:16.234763 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 55330.2us
I20250114 02:54:16.243719 1553 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:16.234774 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:16.236637 1580 raft_consensus.cc:1270] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250114 02:54:16.236521 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 45483us
I20250114 02:54:16.239025 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.246133 1736 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:16.183003 3176 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:16.246680 1702 leader_election.cc:304] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:16.246929 1628 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:16.182215 3123 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:16.245400 1657 raft_consensus.cc:1270] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:16.248767 1628 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:16.250833 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:16.251174 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.196133 3114 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.251885 1736 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:16.253023 1584 raft_consensus.cc:2388] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 5: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:16.192787 3145 raft_consensus.cc:695] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:16.253876 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.256932 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.255102 1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36436 (request call id 759) took 1386 ms. Trace:
I20250114 02:54:16.258476 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.254519 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.259296 1580 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:16.257007 1628 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:16.092994 3057 raft_consensus.cc:513] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.093098 3092 raft_consensus.cc:513] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.262493 3145 consensus_queue.cc:237] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.263773 3086 raft_consensus.cc:2743] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 02:54:16.265759 1653 raft_consensus.cc:1270] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:16.186002 3103 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.266466 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:16.267695 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.238718 3153 raft_consensus.cc:3054] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:16.268723 1580 raft_consensus.cc:1270] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20250114 02:54:16.271970 1580 pending_rounds.cc:77] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 3
I20250114 02:54:16.272184 1580 pending_rounds.cc:99] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250114 02:54:16.272387 1580 raft_consensus.cc:2883] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:16.272672 1657 raft_consensus.cc:1270] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:16.269614 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.186739 3062 raft_consensus.cc:513] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.245782 3150 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.275481 3062 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.276590 3114 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.276935 3139 raft_consensus.cc:3049] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:16.277282 3139 raft_consensus.cc:738] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:16.278801 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.275496 3150 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.279827 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.280541 3150 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.277284 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.281702 1587 raft_consensus.cc:1270] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250114 02:54:16.183264 3068 ts_tablet_manager.cc:1462] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd: Trace:
0114 02:54:12.922233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:12.944177 (+ 21944us) service_pool.cc:225] Handling call
0114 02:54:13.037648 (+ 93471us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0114 02:54:13.037780 (+ 132us) ts_tablet_manager.cc:676] Creating new metadata...
0114 02:54:13.106382 (+ 68602us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:13.417602 (+311220us) tablet_replica.cc:206] Creating consensus instance
0114 02:54:13.537484 (+119882us) inbound_call.cc:173] Queueing success response
0114 02:54:13.620924 (+ 83440us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0114 02:54:14.818018 (+1197094us) ts_tablet_manager.cc:1429] Starting tablet replica
0114 02:54:14.919742 (+101724us) tablet_replica.cc:260] Starting instrumentation
0114 02:54:14.920356 (+ 614us) tablet_replica.cc:279] Starting consensus
Metrics: {"mutex_wait_us":15782,"raft.queue_time_us":62375,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"tablet-open.queue_time_us":186493,"thread_start_us":90531,"threads_started":1}
W20250114 02:54:16.282564 3146 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 1.114s user 0.005s sys 0.007s
W20250114 02:54:16.282896 3146 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 1.188s user 0.005s sys 0.007s
I20250114 02:54:16.283298 3151 raft_consensus.cc:3049] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:16.258863 1727 rpcz_store.cc:276] 0114 02:54:14.862446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.862627 (+ 181us) service_pool.cc:225] Handling call
0114 02:54:14.862990 (+ 363us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:14.889988 (+ 26998us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:14.890032 (+ 44us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:14.913568 (+ 23536us) log.cc:844] Serialized 26 byte log entry
0114 02:54:15.437304 (+523736us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:15.812865 (+375561us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b48001e0258. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbceaf99581 00007fbceaf90677 00007fbceaf9050a 00007fbceb0849bf 00007fbceb084687 00007fbceb098b8c 00007fbceb098b38 00007fbceb098ab8 00007fbceb098a73 00007fbceb097d1c 00007fbcea004d16
0114 02:54:15.913618 (+100753us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x7b18001d0860. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce317e565 00007fbcebd907f9 00007fbce319ba6b 00007fbce3199e6f 00007fbce3199305 00007fbce9aaa107 00007fbce60df8f7 00007fbce60ef00f 00007fbce60eef88
0114 02:54:15.967071 (+ 53453us) log.cc:844] Serialized 14 byte log entry
0114 02:54:15.971575 (+ 4504us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:15.971645 (+ 70us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:16.249081 (+277436us) raft_consensus.cc:1680] finished
0114 02:54:16.249100 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:16.249217 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45844,"prepare.queue_time_us":35893,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":16253,"raft.queue_time_us":35668,"raft.run_cpu_time_us":6499,"raft.run_wall_time_us":919898,"spinlock_wait_cycles":32816128,"thread_start_us":548340,"threads_started":2,"wal-append.queue_time_us":529799}
I20250114 02:54:16.285671 1727 raft_consensus.cc:1270] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:16.185866 2968 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.287955 2968 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.288718 1582 raft_consensus.cc:2363] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:16.186908 3116 raft_consensus.cc:513] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.291936 3116 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.292398 1733 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:16.294891 1733 raft_consensus.cc:1270] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:16.295876 2916 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:16.290431 1582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34938 (request call id 574) took 1705 ms (client timeout 1650 ms). Trace:
W20250114 02:54:16.297127 1582 rpcz_store.cc:269] 0114 02:54:14.584833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.591328 (+ 6495us) service_pool.cc:225] Handling call
0114 02:54:16.290204 (+1698876us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b540012e154. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:16.290336 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3738525952}
I20250114 02:54:16.298439 3074 leader_election.cc:290] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:16.299521 1731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36436 (request call id 760) took 1434 ms (client timeout 1814 ms). Trace:
W20250114 02:54:16.299914 1731 rpcz_store.cc:269] 0114 02:54:14.864752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.864914 (+ 162us) service_pool.cc:225] Handling call
0114 02:54:16.299333 (+1434419us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54002ad854. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:16.299438 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2954275200}
I20250114 02:54:16.300395 3097 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.268402 3090 raft_consensus.cc:2798] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:16.301167 3090 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.301510 3090 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:16.302147 1582 raft_consensus.cc:1235] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:16.304476 1582 raft_consensus.cc:1270] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:16.283733 3151 raft_consensus.cc:738] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:16.299479 3013 leader_election.cc:290] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.186679 3152 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.255542 2688 raft_consensus.cc:2798] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:16.270538 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:16.249043 3176 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:16.253067 3123 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:16.266896 3063 raft_consensus.cc:2798] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:16.252090 3155 raft_consensus.cc:2743] T 0ce7223c99df448483de68f3463ba115 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:16.273355 1553 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
W20250114 02:54:16.324955 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 51710.6us
I20250114 02:54:16.325778 1553 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:16.305687 3075 raft_consensus.cc:2743] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:16.307580 2923 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.327087 3115 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:16.327504 3115 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:16.327873 3115 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:16.323686 1734 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:16.244930 1659 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:16.334949 3115 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.336862 3115 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.326598 1553 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:16.337800 2923 raft_consensus.cc:2743] T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:16.339437 1553 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:16.340248 2923 raft_consensus.cc:2743] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:16.307477 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:16.336791 1734 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:16.342308 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.219208 3178 consensus_queue.cc:260] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.278077 3139 consensus_queue.cc:260] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.349303 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.320379 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.310410 2688 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.130883 3160 consensus_queue.cc:237] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.323968 3176 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:16.344295 3063 raft_consensus.cc:491] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.358421 3063 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:16.358978 2868 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.359443 3139 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:16.343503 3074 raft_consensus.cc:2743] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 02:54:16.362040 1580 raft_consensus.cc:3054] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:16.363512 1553 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:16.364614 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36c50a200b24d17bb9a68ddc9dfa709" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.320883 3151 consensus_queue.cc:260] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.365411 3151 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:16.367316 3137 tablet_bootstrap.cc:492] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1: No bootstrap required, opened a new log
I20250114 02:54:16.367448 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.367835 1654 raft_consensus.cc:1270] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:16.370003 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
W20250114 02:54:16.353415 1702 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.106.65:46625 timed out after 1.650s (SENT) [suppressed 1 similar messages]
I20250114 02:54:16.370783 1654 raft_consensus.cc:1235] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6. Ops: []
I20250114 02:54:16.372020 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:16.367852 3137 ts_tablet_manager.cc:1397] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent bootstrapping tablet: real 1.454s user 0.028s sys 0.039s
I20250114 02:54:16.353477 2688 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:16.372776 1660 raft_consensus.cc:2388] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 7: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:16.374464 1553 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:16.375027 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:16.379343 1654 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:16.379555 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:16.379886 1553 leader_election.cc:400] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6.
I20250114 02:54:16.380182 1553 leader_election.cc:403] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250114 02:54:16.380611 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.381067 1660 raft_consensus.cc:2388] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:16.382391 3152 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.383561 3103 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.386042 1580 raft_consensus.cc:1270] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:16.368803 3178 raft_consensus.cc:3054] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:54:16.373073 2968 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:16.386863 2926 raft_consensus.cc:2798] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:16.353600 3086 consensus_queue.cc:1035] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20250114 02:54:16.405970 1654 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:16.378784 2756 leader_election.cc:290] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.355126 3117 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.408775 1580 raft_consensus.cc:1235] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:16.391613 3083 raft_consensus.cc:2743] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6.
I20250114 02:54:16.428134 3115 raft_consensus.cc:491] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:16.428932 3115 raft_consensus.cc:513] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.431641 3115 leader_election.cc:290] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.393519 3009 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:16.398633 2926 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.438933 1735 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:16.413002 3131 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.451140 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
W20250114 02:54:16.452293 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 98912.5us
I20250114 02:54:16.452407 1580 raft_consensus.cc:3054] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:16.404558 2887 consensus_queue.cc:1035] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.048s
I20250114 02:54:16.453531 3092 leader_election.cc:290] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.442013 2926 raft_consensus.cc:3054] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:16.456579 1702 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:16.413746 3074 consensus_queue.cc:1046] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
I20250114 02:54:16.461838 3117 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:16.465670 3173 log.cc:926] Time spent T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.491s user 0.004s sys 0.000s
W20250114 02:54:16.466822 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 72017.7us
W20250114 02:54:16.469513 3197 log.cc:926] Time spent T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
W20250114 02:54:16.472607 3177 log.cc:926] Time spent T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.450s user 0.000s sys 0.003s
I20250114 02:54:16.464105 3106 tablet_bootstrap.cc:492] T c4ff92fc16244b87b6d30cf38194182d P b89550620e1d455d887be72232bb5454: No bootstrap required, opened a new log
I20250114 02:54:16.475540 3106 ts_tablet_manager.cc:1397] T c4ff92fc16244b87b6d30cf38194182d P b89550620e1d455d887be72232bb5454: Time spent bootstrapping tablet: real 1.906s user 0.042s sys 0.021s
I20250114 02:54:16.442759 3009 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.476362 1702 leader_election.cc:400] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 5. Current term is 6.
I20250114 02:54:16.476588 1702 leader_election.cc:403] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250114 02:54:16.457944 3057 leader_election.cc:290] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.464612 3152 raft_consensus.cc:2743] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:16.485360 1553 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:16.491065 1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48632 (request call id 644) took 1510 ms. Trace:
I20250114 02:54:16.482817 3075 raft_consensus.cc:2743] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 5. Current term is 6.
I20250114 02:54:16.492630 3080 consensus_queue.cc:1035] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250114 02:54:16.493168 3203 raft_consensus.cc:2798] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:16.493934 3203 raft_consensus.cc:695] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:16.491530 1661 rpcz_store.cc:276] 0114 02:54:14.978717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.978858 (+ 141us) service_pool.cc:225] Handling call
0114 02:54:15.073513 (+ 94655us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:15.429620 (+356107us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:15.429665 (+ 45us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:15.441074 (+ 11409us) log.cc:844] Serialized 26 byte log entry
0114 02:54:15.756409 (+315335us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:15.756460 (+ 51us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:16.094977 (+338517us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b54003c6ed4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:16.095038 (+ 61us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:16.465388 (+370350us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b24001be4f8. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce2ff39d4 00007fbce2ff1adb 00007fbce3080289 00007fbce307ff15 00007fbce307f55d 00007fbce5d3bd3e 00007fbce5d534cc 00007fbce5d527f6 00007fbce5d52418 00007fbce5d5bdf1
0114 02:54:16.488806 (+ 23418us) raft_consensus.cc:1680] finished
0114 02:54:16.488819 (+ 13us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:16.488918 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24282,"raft.queue_time_us":5526,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":8044,"spinlock_wait_cycles":848470656,"thread_start_us":281181,"threads_started":1,"wal-append.queue_time_us":499194}
W20250114 02:54:16.498013 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 189662us
I20250114 02:54:16.479398 3083 consensus_queue.cc:1035] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20250114 02:54:16.500198 3131 raft_consensus.cc:3049] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Stepping down as leader of term 5
I20250114 02:54:16.509418 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.509721 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.509455 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.495457 3090 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.488906 3117 consensus_queue.cc:1035] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
W20250114 02:54:16.513545 1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36452 (request call id 574) took 1201 ms (client timeout 1562 ms). Trace:
I20250114 02:54:16.513674 1733 raft_consensus.cc:1235] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
W20250114 02:54:16.511723 1613 tablet.cc:2367] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 02:54:16.494657 3153 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.516166 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ff92fc16244b87b6d30cf38194182d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:16.513911 1730 rpcz_store.cc:269] 0114 02:54:15.312002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.313410 (+ 1408us) service_pool.cc:225] Handling call
0114 02:54:16.513386 (+1199976us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b54000e0bd4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d92ae 00007fbceb033c17 00007fbce4701c14 00007fbce4701ba2 00007fbce4701ad1 00007fbce4701a4c 00007fbce4700d02 00007fbce4500dfc 00007fbce4500236 00007fbce44ffbdf
0114 02:54:16.513487 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86401920}
I20250114 02:54:16.490295 3063 raft_consensus.cc:513] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.519104 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:16.527391 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.519358 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:16.520926 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.521232 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.528465 1654 raft_consensus.cc:2363] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:16.528746 1587 raft_consensus.cc:2405] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 02:54:16.522400 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.503371 3154 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.527997 1580 raft_consensus.cc:2463] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:16.529615 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.523026 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.529238 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.529187 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:16.531340 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.531711 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:16.523893 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf552d03bce48dda4e5b9772287649a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.533030 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:16.534799 1581 raft_consensus.cc:2363] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:16.500308 3155 consensus_queue.cc:1035] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20250114 02:54:16.534034 1580 raft_consensus.cc:3054] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:16.537478 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.539656 1657 raft_consensus.cc:2388] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 5: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:16.536953 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.540731 1657 raft_consensus.cc:1270] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:16.540961 1655 raft_consensus.cc:2405] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 02:54:16.540405 2926 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.515648 2843 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.542737 2926 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.543100 3009 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625)
I20250114 02:54:16.543862 3137 raft_consensus.cc:357] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.544458 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.545146 1584 raft_consensus.cc:2363] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:16.546481 1628 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:16.534750 3154 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.548048 1628 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:16.548487 3154 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.549027 3154 raft_consensus.cc:2798] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:16.549165 1628 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:16.549393 3154 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.549732 3154 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:16.549923 3149 raft_consensus.cc:2798] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:16.550207 1628 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:16.550768 2971 raft_consensus.cc:2743] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 02:54:16.551297 2885 raft_consensus.cc:2798] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:16.551622 2885 raft_consensus.cc:491] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.551925 2885 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:16.511287 3131 raft_consensus.cc:738] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:16.554919 1660 raft_consensus.cc:1270] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
@ 0x7fbce2826950 google::LogMessage::Flush() at ??:0
I20250114 02:54:16.544641 3137 raft_consensus.cc:383] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:16.516007 3145 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.539009 1653 raft_consensus.cc:2388] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 3: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:16.558454 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.558668 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:16.558501 1662 raft_consensus.cc:1235] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:16.559309 1736 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:16.554467 1579 raft_consensus.cc:2388] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 6: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:16.562474 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:16.563035 1734 raft_consensus.cc:3054] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Advancing to term 3
I20250114 02:54:16.563021 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:16.569828 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:16.572387 1736 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 7.
I20250114 02:54:16.550277 3149 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:16.582064 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.583431 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ff92fc16244b87b6d30cf38194182d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.583840 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d291f7c9c5547cf9006ed2dcc6b048e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.632164 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.535398 3072 leader_election.cc:290] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.634915 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bae187c925c49dd9bb36f34be387836" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:16.635510 1730 raft_consensus.cc:3054] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:16.645012 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aa8b25b08141db9f0576ce3fa34d3c" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.494758 3203 consensus_queue.cc:237] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.556814 3137 raft_consensus.cc:738] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Initialized, Role: FOLLOWER
I20250114 02:54:16.534860 3081 consensus_queue.cc:1035] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.101s
I20250114 02:54:16.636092 3149 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
W20250114 02:54:16.644407 3201 log.cc:926] Time spent T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20250114 02:54:16.679819 1734 raft_consensus.cc:2463] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 3.
I20250114 02:54:16.573660 3145 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.686848 1702 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:16.688319 1702 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:16.695151 2868 consensus_queue.cc:1035] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:16.696501 2916 consensus_queue.cc:1035] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:16.686174 1730 raft_consensus.cc:2463] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:16.698045 2688 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.699862 3106 raft_consensus.cc:357] T c4ff92fc16244b87b6d30cf38194182d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.700462 2688 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.646984 3160 consensus_queue.cc:1046] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20250114 02:54:16.700933 3106 raft_consensus.cc:383] T c4ff92fc16244b87b6d30cf38194182d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:16.701380 3106 raft_consensus.cc:738] T c4ff92fc16244b87b6d30cf38194182d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Initialized, Role: FOLLOWER
I20250114 02:54:16.703078 3145 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.712195 3154 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.713950 1661 raft_consensus.cc:2363] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 5. Current term is 6.
I20250114 02:54:16.715864 3154 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.718878 3137 consensus_queue.cc:260] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.721467 3137 ts_tablet_manager.cc:1428] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Time spent starting tablet: real 0.337s user 0.005s sys 0.003s
I20250114 02:54:16.724071 3090 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.726641 3077 leader_election.cc:290] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.728875 3063 leader_election.cc:290] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.730069 3153 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.736564 2968 raft_consensus.cc:2743] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:16.738730 3175 raft_consensus.cc:2743] T 5e00ae680c2d4c5cbb5154723228286f P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:16.681594 3131 consensus_queue.cc:260] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.741636 3153 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.741793 3146 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.742110 3153 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.742341 3146 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.744009 3153 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.744213 3146 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.744967 2878 raft_consensus.cc:491] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.745324 3117 raft_consensus.cc:2743] T ee328440ef8f409ca059ac947d40a58a P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:16.746555 1654 raft_consensus.cc:2388] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:16.747231 3123 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.745396 2878 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:16.751392 2843 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.752491 1729 raft_consensus.cc:1270] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:16.755005 1702 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:16.756364 1702 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:16.757745 1702 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
W20250114 02:54:16.759399 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 110631us
I20250114 02:54:16.767102 1628 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:16.774837 3009 raft_consensus.cc:2743] T 2d291f7c9c5547cf9006ed2dcc6b048e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:16.776211 1702 leader_election.cc:304] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:16.778172 1583 raft_consensus.cc:1235] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:16.776645 3103 raft_consensus.cc:2798] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:16.768537 3175 consensus_queue.cc:1035] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20250114 02:54:16.779412 3103 raft_consensus.cc:695] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:16.780248 3103 consensus_queue.cc:237] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.762741 3117 raft_consensus.cc:2743] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:16.768546 3146 consensus_queue.cc:1035] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20250114 02:54:16.787796 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.787711 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.788679 1579 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:16.790354 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.792671 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:16.793354 1584 raft_consensus.cc:2388] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 7: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:16.793958 1587 raft_consensus.cc:1235] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:16.794545 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:16.794919 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.795207 1587 raft_consensus.cc:2388] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 6: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:16.795099 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.795701 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:16.796444 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:16.796770 1582 raft_consensus.cc:3054] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:16.797205 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.797603 1581 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:16.798130 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:16.798888 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:16.799547 1736 raft_consensus.cc:2388] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 7: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:16.776966 3013 raft_consensus.cc:2743] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:16.801553 1581 raft_consensus.cc:2388] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:16.801414 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:16.805285 1579 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:16.802630 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:16.800855 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:16.805939 1584 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:16.803303 1702 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:16.801410 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.806655 1735 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:16.702230 3106 consensus_queue.cc:260] T c4ff92fc16244b87b6d30cf38194182d P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.808063 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:16.808676 1734 raft_consensus.cc:3054] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:16.809350 1580 raft_consensus.cc:2463] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:16.803633 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:54:16.786427 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 264820us
I20250114 02:54:16.786713 3075 raft_consensus.cc:2743] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 02:54:16.812204 1727 raft_consensus.cc:2388] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 3: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:16.814147 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:16.814217 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b78459d23a744819d55e5f50de8174f" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:16.814935 1730 raft_consensus.cc:2388] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 6: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:16.814080 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:16.816176 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b972e53ce14bd0825dc78f9c5c1a27" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:16.804589 3072 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:16.823967 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.830016 1734 raft_consensus.cc:2463] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:16.830893 1579 raft_consensus.cc:2463] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:16.832805 1702 leader_election.cc:400] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:16.833184 1702 leader_election.cc:403] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250114 02:54:16.817704 3153 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.800271 3117 consensus_queue.cc:1035] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20250114 02:54:16.819307 2796 raft_consensus.cc:2798] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250114 02:54:16.834235 1553 leader_election.cc:343] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:16.833072 3072 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.831132 2968 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.844090 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:16.845642 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:16.843562 3152 raft_consensus.cc:2743] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:16.847736 1655 raft_consensus.cc:3054] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:16.821911 3122 consensus_queue.cc:1035] T 30486d50dbd94c78a14bad5b98494276 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.019s
I20250114 02:54:16.855849 3072 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.837556 2796 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:16.837613 2916 raft_consensus.cc:491] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:16.860694 1582 raft_consensus.cc:2463] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:16.858402 1655 raft_consensus.cc:2463] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:16.863327 1553 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:16.867993 3086 raft_consensus.cc:2798] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:16.868530 3086 raft_consensus.cc:695] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:16.858402 3131 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:16.869649 3086 consensus_queue.cc:237] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.871989 3106 ts_tablet_manager.cc:1428] T c4ff92fc16244b87b6d30cf38194182d P b89550620e1d455d887be72232bb5454: Time spent starting tablet: real 0.396s user 0.008s sys 0.001s
W20250114 02:54:16.877089 3176 consensus_meta.cc:287] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.509s user 0.005s sys 0.006s
I20250114 02:54:16.877506 3176 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.837345 2885 raft_consensus.cc:513] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.865751 2916 raft_consensus.cc:513] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.879928 2885 leader_election.cc:290] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.838678 3153 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.880482 2916 leader_election.cc:290] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.848276 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:16.881647 1659 raft_consensus.cc:2388] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:16.881927 3153 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.882789 3140 raft_consensus.cc:3049] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:16.883298 3140 raft_consensus.cc:738] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:16.884290 3140 consensus_queue.cc:260] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.858937 3123 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.887348 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.887667 2878 raft_consensus.cc:513] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.888734 1732 raft_consensus.cc:2388] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:16.888861 3140 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:16.848631 3160 raft_consensus.cc:491] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.891486 3160 raft_consensus.cc:513] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.893536 3160 leader_election.cc:290] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.894129 3129 leader_election.cc:290] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:16.896342 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 47488.1us
I20250114 02:54:16.862429 2796 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:16.848647 2968 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.906780 3145 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.937729 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48636 (request call id 599) took 1007 ms. Trace:
I20250114 02:54:16.938247 1656 rpcz_store.cc:276] 0114 02:54:15.930221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.930428 (+ 207us) service_pool.cc:225] Handling call
0114 02:54:15.941759 (+ 11331us) raft_consensus.cc:1466] Updating replica for 0 ops
0114 02:54:16.213332 (+271573us) raft_consensus.cc:1529] Early marking committed up to index 3
0114 02:54:16.217765 (+ 4433us) log.cc:844] Serialized 14 byte log entry
0114 02:54:16.500666 (+282901us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0114 02:54:16.500714 (+ 48us) raft_consensus.cc:1640] Marking committed up to 3
0114 02:54:16.500749 (+ 35us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:16.937434 (+436685us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b5400017f54. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d5f51 00007fbce60d499e 00007fbceb0332d9 00007fbce46ff694 00007fbce46ff622 00007fbce46ff551 00007fbce46ff4cc 00007fbce46fe782 00007fbce4500dfc 00007fbce4500236
0114 02:54:16.937497 (+ 63us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:16.937660 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":178531,"prepare.queue_time_us":4694,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":153991,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":3096,"spinlock_wait_cycles":624176896,"thread_start_us":129898,"threads_started":1,"wal-append.queue_time_us":175390}
I20250114 02:54:16.940675 1628 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:16.945900 3150 raft_consensus.cc:2743] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:16.944283 1628 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:16.947109 3150 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:16.947564 3150 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.947980 3150 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:16.955341 1628 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:16.961354 3149 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.939421 3145 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.968915 3150 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.969966 3145 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.932997 3075 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:16.971542 3075 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:16.973511 3150 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:16.973845 3075 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.933851 3080 raft_consensus.cc:491] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:16.985901 1628 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:16.987223 1654 raft_consensus.cc:1235] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5. Ops: []
W20250114 02:54:16.985006 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 137799us
I20250114 02:54:16.934305 3176 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:16.988941 1702 leader_election.cc:400] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:16.990875 1702 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:16.995662 1628 leader_election.cc:304] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:16.983704 3209 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:16.996471 1628 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:16.997103 2710 raft_consensus.cc:2798] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:16.998495 1553 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.009440 1579 raft_consensus.cc:1270] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:17.058413 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:17.009820 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:17.046729 1658 raft_consensus.cc:1235] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:17.060060 1702 leader_election.cc:400] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:17.060940 1702 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:17.061971 1702 leader_election.cc:304] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:17.059232 1654 raft_consensus.cc:2388] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:17.062270 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:17.026584 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bb7ee74c418aabd9237a6f6a0454" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:17.022858 1586 raft_consensus.cc:1235] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:16.998989 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:17.065780 1582 raft_consensus.cc:2388] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:17.069907 1702 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:17.072206 1702 leader_election.cc:304] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:16.995157 3075 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
W20250114 02:54:17.075314 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20250114 02:54:17.074863 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 78437.1us
I20250114 02:54:17.084300 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:17.085119 1736 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:17.086434 1733 raft_consensus.cc:1270] T e36c50a200b24d17bb9a68ddc9dfa709 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 02:54:17.087677 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:17.088336 1734 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:17.090739 1735 raft_consensus.cc:1270] T 4a25a35a51714764a2d368283f31d19a P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:17.094130 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:17.094810 1735 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:17.095952 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:17.097921 2878 leader_election.cc:290] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:17.098079 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ff92fc16244b87b6d30cf38194182d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:17.098682 1728 raft_consensus.cc:2463] T c4ff92fc16244b87b6d30cf38194182d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:54:17.099159 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:16.989071 3145 consensus_queue.cc:1046] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
I20250114 02:54:16.989892 3080 raft_consensus.cc:513] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:17.103335 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 104926us
I20250114 02:54:17.040642 3150 raft_consensus.cc:2798] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:17.053073 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:17.053723 1578 raft_consensus.cc:1235] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250114 02:54:17.104480 1628 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.103677 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.047684 3209 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:17.040313 3160 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:17.010967 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:17.108355 1702 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:17.102501 3080 leader_election.cc:290] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:17.112267 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdaf3f35acd45469c23cec4d4112b11" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.112870 1580 raft_consensus.cc:3054] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:17.098623 3122 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:17.115017 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:17.105937 1656 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:17.117396 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:17.118598 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:17.119148 1662 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 6.
I20250114 02:54:17.119508 3155 consensus_queue.cc:1035] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:54:17.115236 3160 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:17.120523 3149 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.120760 3160 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:17.121856 1659 raft_consensus.cc:1270] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20250114 02:54:17.122182 1659 pending_rounds.cc:77] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Aborting all ops after (but not including) 3
I20250114 02:54:17.122416 1659 pending_rounds.cc:99] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20250114 02:54:17.070734 3215 raft_consensus.cc:2743] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:17.122687 1659 raft_consensus.cc:2883] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:17.125036 1580 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:17.125775 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.126411 1580 raft_consensus.cc:2388] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 5: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:17.115407 3209 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:17.127065 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:17.128010 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.128880 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:17.129410 1580 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:17.115610 2878 raft_consensus.cc:2743] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:17.131155 1578 raft_consensus.cc:1270] T edafd45551ec43f9a5b750c1e54c62f6 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:17.133136 1702 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:16.997665 2968 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.138890 1628 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:17.139022 1553 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.138936 1579 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:17.076547 3176 raft_consensus.cc:2743] T fb007689a5d64ee2a8fa67adad56de19 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:17.074712 3152 raft_consensus.cc:2743] T daf552d03bce48dda4e5b9772287649a P b89550620e1d455d887be72232bb5454 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:17.144619 1734 raft_consensus.cc:2463] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:17.148521 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ff92fc16244b87b6d30cf38194182d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:17.083985 2710 raft_consensus.cc:695] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:17.073871 3075 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.149245 1582 raft_consensus.cc:2463] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 0.
I20250114 02:54:17.105162 2885 raft_consensus.cc:2798] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:17.153153 1731 raft_consensus.cc:1270] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:17.118999 3203 raft_consensus.cc:2743] T 464b2086919241de965a1ac17809451c P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:17.158972 1553 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:17.162319 1702 leader_election.cc:400] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 5. Current term is 6.
I20250114 02:54:17.162350 1628 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.159942 3215 raft_consensus.cc:2798] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:17.162868 1702 leader_election.cc:403] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250114 02:54:17.166726 1553 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.114892 3122 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.168095 1628 leader_election.cc:304] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
W20250114 02:54:17.171386 1553 leader_election.cc:343] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:17.171806 1553 leader_election.cc:304] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.150336 3080 raft_consensus.cc:2743] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:17.155036 3154 raft_consensus.cc:2743] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 02:54:17.173004 1553 leader_election.cc:304] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.211735 1553 leader_election.cc:304] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.212577 1553 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.150038 3081 raft_consensus.cc:2743] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 02:54:17.209702 1628 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.216312 1553 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:17.161381 3150 raft_consensus.cc:695] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:17.150149 2843 consensus_queue.cc:1046] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.030s
I20250114 02:54:17.236281 1628 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:17.169453 3149 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:17.166112 3215 raft_consensus.cc:491] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:17.172292 3063 raft_consensus.cc:2743] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 02:54:17.169026 3146 consensus_queue.cc:1046] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.083s
I20250114 02:54:17.249346 3149 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:17.173501 3072 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election won for term 7
I20250114 02:54:17.248523 1553 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:17.200024 3160 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.255071 3153 raft_consensus.cc:2798] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:17.256955 3160 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.258076 1653 raft_consensus.cc:1235] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250114 02:54:17.170215 2885 raft_consensus.cc:695] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:17.264057 2885 consensus_queue.cc:237] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.159970 2878 consensus_queue.cc:1035] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
I20250114 02:54:17.265735 3176 consensus_queue.cc:1046] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:17.266758 2878 raft_consensus.cc:3049] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:17.267103 2878 raft_consensus.cc:738] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:17.267892 2878 consensus_queue.cc:260] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.269163 3075 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:17.270607 2710 consensus_queue.cc:237] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.271801 3209 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.173270 3115 raft_consensus.cc:2743] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:17.208793 3086 raft_consensus.cc:2743] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:17.222806 2971 raft_consensus.cc:2743] T 0b78459d23a744819d55e5f50de8174f P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:17.222210 3077 raft_consensus.cc:2743] T 01ce953d409541bb85818e3dcf36541b P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:17.223603 3203 raft_consensus.cc:2743] T 2d291f7c9c5547cf9006ed2dcc6b048e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:17.173810 3175 raft_consensus.cc:2743] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 5. Current term is 6.
I20250114 02:54:17.221856 3123 raft_consensus.cc:2743] T b4aa8b25b08141db9f0576ce3fa34d3c P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:17.247819 3145 raft_consensus.cc:2743] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 02:54:17.251731 3081 raft_consensus.cc:2798] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:54:17.288419 1628 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:17.251140 3154 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:17.292212 1628 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:17.213610 2796 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.251363 3215 raft_consensus.cc:3054] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:17.278659 2843 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:17.298246 2843 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:17.273576 3097 consensus_queue.cc:1035] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20250114 02:54:17.313143 2843 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Advancing to term 7
W20250114 02:54:17.315285 3214 log.cc:926] Time spent T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20250114 02:54:17.318665 3129 consensus_queue.cc:1035] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:54:17.322921 1727 raft_consensus.cc:1235] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:17.294206 3160 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:17.337074 3081 raft_consensus.cc:695] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:17.359594 3081 consensus_queue.cc:237] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.310101 3145 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:17.288355 3149 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
W20250114 02:54:17.365928 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 73807.3us
I20250114 02:54:17.367157 1628 leader_election.cc:304] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:17.301741 3103 consensus_queue.cc:1035] T e36c50a200b24d17bb9a68ddc9dfa709 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
W20250114 02:54:17.371492 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 156307us
I20250114 02:54:17.373732 1702 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:17.304409 3063 consensus_queue.cc:1035] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.062s
I20250114 02:54:17.295209 3154 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.338402 3209 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.337752 3160 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:17.380548 1628 leader_election.cc:304] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:17.367020 3145 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:17.374501 2868 raft_consensus.cc:2798] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:17.369568 3153 raft_consensus.cc:695] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:17.377250 3140 raft_consensus.cc:2798] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:17.381124 3160 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:17.310302 3150 consensus_queue.cc:237] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.384872 3145 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:17.400120 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd7d01e627ea4bfc86adf4c4fa3f6139" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:17.400790 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee328440ef8f409ca059ac947d40a58a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:17.387486 3122 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.389171 3140 raft_consensus.cc:491] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:17.387149 2868 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:17.418174 2868 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:17.425010 3153 consensus_queue.cc:237] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.425557 3215 raft_consensus.cc:513] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.427445 3215 leader_election.cc:290] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:17.409466 3140 raft_consensus.cc:3054] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.409562 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
W20250114 02:54:17.415035 3221 log.cc:926] Time spent T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20250114 02:54:17.416262 3154 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.417621 2796 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:17.407366 3209 raft_consensus.cc:2798] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:17.429512 3122 raft_consensus.cc:491] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:17.431134 3178 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:17.441659 3092 raft_consensus.cc:3049] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:17.459757 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:17.460976 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:17.461678 1734 raft_consensus.cc:2371] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4. Re-sending same reply.
I20250114 02:54:17.441443 3072 raft_consensus.cc:695] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:17.460999 1736 raft_consensus.cc:2388] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:17.470012 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
W20250114 02:54:17.472507 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 92059.5us
W20250114 02:54:17.474676 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 187255us
I20250114 02:54:17.475832 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb007689a5d64ee2a8fa67adad56de19" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:17.479540 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:17.480147 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.479607 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.455554 3209 raft_consensus.cc:695] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:17.459728 3092 raft_consensus.cc:738] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:17.493355 1655 raft_consensus.cc:1270] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:17.493767 3072 consensus_queue.cc:237] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.494163 3233 consensus_queue.cc:1046] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:17.462646 3178 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.496055 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.497296 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c818f667d1314fc68ee42a2fd1c5a627" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:17.499336 1585 raft_consensus.cc:1235] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 5. Current term is 6. Ops: []
I20250114 02:54:17.499403 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:17.499482 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:17.500337 1735 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 6.
I20250114 02:54:17.503479 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.504917 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:17.505885 1654 raft_consensus.cc:2388] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:17.506917 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:17.507511 1584 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:17.508708 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:17.523180 1578 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:17.518821 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.524106 1579 raft_consensus.cc:2388] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 6: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:17.509915 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:17.509374 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:17.525701 1736 raft_consensus.cc:1270] T 6bae187c925c49dd9bb36f34be387836 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:17.510354 1660 raft_consensus.cc:1235] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250114 02:54:17.459991 3122 raft_consensus.cc:513] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.533160 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39189f5c30e4eec95701fe6a370896e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:17.536948 1628 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.545501 1658 raft_consensus.cc:1235] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:17.545833 1662 raft_consensus.cc:1270] T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250114 02:54:17.546772 1657 raft_consensus.cc:1235] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250114 02:54:17.549309 1655 raft_consensus.cc:1270] T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:17.544111 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:17.550884 1654 raft_consensus.cc:2405] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 02:54:17.538166 1702 leader_election.cc:304] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:17.492236 3090 consensus_queue.cc:1035] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.057s
I20250114 02:54:17.558372 3146 raft_consensus.cc:2743] T be63bb7ee74c418aabd9237a6f6a0454 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:17.554908 2796 consensus_queue.cc:1035] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20250114 02:54:17.574219 1729 raft_consensus.cc:1270] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 02:54:17.554284 1702 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:17.597613 1702 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:17.561779 1553 leader_election.cc:304] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.600623 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:17.600970 1553 leader_election.cc:304] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.512580 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:17.599946 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:17.603511 1702 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
W20250114 02:54:17.571056 3222 log.cc:926] Time spent T e36c50a200b24d17bb9a68ddc9dfa709 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.210s user 0.003s sys 0.000s
@ 0x7fbce282ba8b google::LogMessageFatal::~LogMessageFatal() at ??:0
W20250114 02:54:17.608637 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 71790.7us
I20250114 02:54:17.572641 3090 consensus_queue.cc:1225] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer b89550620e1d455d887be72232bb5454 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250114 02:54:17.572579 3122 leader_election.cc:290] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.587356 2868 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.614735 3081 raft_consensus.cc:2743] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:17.603178 2710 consensus_queue.cc:1046] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20250114 02:54:17.557026 3092 consensus_queue.cc:260] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.557590 3178 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.589165 3160 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.580359 2878 raft_consensus.cc:3054] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:17.584352 2843 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.621680 1654 raft_consensus.cc:2463] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:17.632876 2843 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.567334 3154 raft_consensus.cc:2743] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 02:54:17.618912 2756 raft_consensus.cc:3049] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 02:54:17.619751 2916 raft_consensus.cc:3049] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:17.576434 3149 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.606400 2796 raft_consensus.cc:2743] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:17.578734 3145 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.622727 1482 catalog_manager.cc:5526] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 2, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 2 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:17.626082 3090 consensus_queue.cc:1035] T edafd45551ec43f9a5b750c1e54c62f6 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.010s
I20250114 02:54:17.630764 3160 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.630797 3228 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:17.631040 1660 raft_consensus.cc:1235] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6. Ops: []
W20250114 02:54:17.623968 3239 log.cc:926] Time spent T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20250114 02:54:17.681243 1731 raft_consensus.cc:2405] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20250114 02:54:17.678102 2885 consensus_queue.cc:1046] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20250114 02:54:17.684749 3145 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.685096 3160 raft_consensus.cc:3049] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 02:54:17.685428 3160 raft_consensus.cc:738] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:17.686190 3160 consensus_queue.cc:260] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.687443 3160 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:17.638198 1553 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:17.605479 3234 raft_consensus.cc:2743] T ee328440ef8f409ca059ac947d40a58a P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:17.656540 1628 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:17.654472 3227 raft_consensus.cc:2743] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:17.653389 2916 raft_consensus.cc:738] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:17.583632 3209 consensus_queue.cc:237] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.714886 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.646087 2756 raft_consensus.cc:738] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:17.646647 3140 raft_consensus.cc:513] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.660929 1702 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:17.661491 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:17.663620 3149 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.668105 2971 raft_consensus.cc:3049] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:17.669548 2868 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625)
I20250114 02:54:17.674495 1482 catalog_manager.cc:5526] T 8c49b6c0ab1d49079166b6a4cdf59896 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
W20250114 02:54:17.695340 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 57237.3us
I20250114 02:54:17.706115 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:17.708349 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.711900 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:17.728916 1735 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:17.714082 1584 raft_consensus.cc:1270] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:17.713943 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.717049 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:17.714356 3129 raft_consensus.cc:2798] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:17.718250 1729 pending_rounds.cc:77] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 1
I20250114 02:54:17.730216 1733 raft_consensus.cc:3054] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:17.718685 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:17.719233 1580 raft_consensus.cc:1270] T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 02:54:17.717548 1583 raft_consensus.cc:2388] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 7: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:17.730818 1728 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:17.722344 1628 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.728235 1482 catalog_manager.cc:5526] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 3 to 4, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 4 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
W20250114 02:54:17.713943 3251 log.cc:926] Time spent T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20250114 02:54:17.730501 1729 pending_rounds.cc:99] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 02:54:17.731943 1729 raft_consensus.cc:2883] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:17.732520 1482 catalog_manager.cc:5526] T 5d77d39f9a4b4a57ae17db8e50237cdb P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 2 to 3, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 3 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:17.734149 1578 raft_consensus.cc:1235] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 3. Current term is 4. Ops: []
I20250114 02:54:17.735291 1482 catalog_manager.cc:5526] T 8d4d46461bf745639ea4f64fa3bcb996 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:17.736941 1482 catalog_manager.cc:5526] T 2d291f7c9c5547cf9006ed2dcc6b048e P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:17.739145 1733 raft_consensus.cc:2463] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 7.
I20250114 02:54:17.738801 1482 catalog_manager.cc:5526] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 4 to 5, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 5 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:17.694837 3228 raft_consensus.cc:695] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:17.740366 1628 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.740808 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:17.743057 1628 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.744848 1553 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.746191 1553 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.741225 1482 catalog_manager.cc:5526] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 3 to 4, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 4 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:17.714843 3103 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:17.748378 1482 catalog_manager.cc:5526] T ef83c2f4c7494118a6564eaca6b7799d P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 2 to 4, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 4 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:17.752691 1553 leader_election.cc:304] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.754971 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:17.756789 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:17.757267 1661 raft_consensus.cc:2388] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 7: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:17.758215 1654 raft_consensus.cc:1235] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250114 02:54:17.751423 1482 catalog_manager.cc:5526] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 4 to 5, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 5 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:17.725314 3154 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:17.759169 3237 consensus_queue.cc:1035] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20250114 02:54:17.727272 2971 raft_consensus.cc:738] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:17.768756 3228 consensus_queue.cc:237] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.783864 1735 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:17.749431 3129 raft_consensus.cc:695] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:17.741822 3145 raft_consensus.cc:2743] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:17.760792 1482 catalog_manager.cc:5526] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:17.743961 3238 raft_consensus.cc:2743] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 02:54:17.749481 3248 raft_consensus.cc:2743] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 02:54:17.784857 3122 raft_consensus.cc:2743] T fb007689a5d64ee2a8fa67adad56de19 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:17.749368 3234 raft_consensus.cc:2743] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
@ 0x7fbcec5d446e kudu::TestWorkload::Setup() at ??:0
I20250114 02:54:17.751214 3103 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:17.810330 3103 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Advancing to term 7
W20250114 02:54:17.811498 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 150634us
I20250114 02:54:17.812460 3147 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:17.812815 3147 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:17.813166 3147 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:17.760164 3081 raft_consensus.cc:2743] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
@ 0x34e988 kudu::master::LeaderRebalancerTest::CreateWorkloadTable() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_leader_rebalancer-test.cc:109
I20250114 02:54:17.801226 1482 catalog_manager.cc:5526] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:17.789520 3154 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.822777 3154 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.807777 3140 leader_election.cc:290] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:17.820072 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.818948 1482 catalog_manager.cc:5526] T fb007689a5d64ee2a8fa67adad56de19 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
@ 0x34b3bf kudu::master::LeaderRebalancerTest_AddTserver_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_leader_rebalancer-test.cc:411
I20250114 02:54:17.825139 1482 catalog_manager.cc:5526] T 6b73d9b5d01245f8be76a02efe7e5eca P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 2 to 5, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 5 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:17.825734 1583 raft_consensus.cc:2388] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:17.830104 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a35a51714764a2d368283f31d19a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:17.830386 1482 catalog_manager.cc:5526] T be63bb7ee74c418aabd9237a6f6a0454 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 3 to 4, leader changed from b89550620e1d455d887be72232bb5454 (127.0.106.67) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 4 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:17.830989 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:17.831947 1727 raft_consensus.cc:2388] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 7: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:17.829092 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9dac23e1497286938e63aa482439" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:17.832162 1482 catalog_manager.cc:5526] T cd7d01e627ea4bfc86adf4c4fa3f6139 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:17.833385 1735 raft_consensus.cc:2405] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 02:54:17.789330 2923 consensus_queue.cc:1046] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.036s
I20250114 02:54:17.836831 1732 raft_consensus.cc:1235] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250114 02:54:17.836750 3147 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.784469 2756 consensus_queue.cc:260] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.843828 2756 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
W20250114 02:54:17.849885 2688 consensus_queue.cc:1159] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b89550620e1d455d887be72232bb5454" responder_term: 3 status { last_received { term: 3 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 3 index: 2 } }
I20250114 02:54:17.788731 2916 consensus_queue.cc:260] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.797797 3232 consensus_queue.cc:1035] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20250114 02:54:17.852177 2916 raft_consensus.cc:3054] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:17.855412 1728 raft_consensus.cc:1270] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250114 02:54:17.855844 1728 pending_rounds.cc:77] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 1
I20250114 02:54:17.856087 1728 pending_rounds.cc:99] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250114 02:54:17.856406 1728 raft_consensus.cc:2883] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:17.834545 1482 catalog_manager.cc:5526] T 7cd3f8af715b4ed8a97965a52b089a21 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 3 to 4, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 4 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:17.826578 2971 consensus_queue.cc:260] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.858209 1482 catalog_manager.cc:5526] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 2 to 3, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66). New cstate: current_term: 3 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:17.859845 1482 catalog_manager.cc:5526] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
W20250114 02:54:17.861276 3256 log.cc:926] Time spent T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20250114 02:54:17.794836 3086 consensus_queue.cc:1035] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20250114 02:54:17.865553 1702 leader_election.cc:304] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:17.867076 2868 consensus_queue.cc:1035] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:54:17.867376 1702 leader_election.cc:304] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:17.868187 3114 raft_consensus.cc:2743] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:17.868811 2868 consensus_queue.cc:1046] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:54:17.869241 3176 raft_consensus.cc:2743] T c39189f5c30e4eec95701fe6a370896e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:17.869653 2868 raft_consensus.cc:3049] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 02:54:17.870000 2868 raft_consensus.cc:738] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:17.870769 3227 consensus_queue.cc:1046] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:17.871487 3227 raft_consensus.cc:3049] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 02:54:17.871762 3227 raft_consensus.cc:738] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:17.872073 3129 consensus_queue.cc:237] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.872416 3227 consensus_queue.cc:260] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.873448 3227 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:17.868757 3114 consensus_queue.cc:1046] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20250114 02:54:17.875309 3114 raft_consensus.cc:3049] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 02:54:17.875649 3114 raft_consensus.cc:738] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:17.876384 3114 consensus_queue.cc:260] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.876889 3235 consensus_queue.cc:1046] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:54:17.877566 3114 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:17.877768 3235 consensus_queue.cc:1046] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:54:17.878490 3123 raft_consensus.cc:3049] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Stepping down as leader of term 5
I20250114 02:54:17.878937 3123 raft_consensus.cc:738] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:17.879647 3123 consensus_queue.cc:260] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.880707 3123 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:17.878531 3235 raft_consensus.cc:3049] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Stepping down as leader of term 5
I20250114 02:54:17.882987 1702 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
W20250114 02:54:17.883183 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 45352.8us
W20250114 02:54:17.885004 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 115931us
I20250114 02:54:17.888413 1702 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:17.888512 1628 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:17.889693 1628 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:17.895995 1553 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.884582 3235 raft_consensus.cc:738] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:17.924881 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:17.842093 3103 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.925983 1628 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:17.890439 3176 raft_consensus.cc:2743] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250114 02:54:17.931509 1553 leader_election.cc:304] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.945187 1628 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:17.959663 1660 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:17.927276 1661 raft_consensus.cc:1235] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250114 02:54:17.961560 1584 raft_consensus.cc:1270] T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:17.902104 3237 raft_consensus.cc:2798] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:17.959033 1580 raft_consensus.cc:1270] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 02:54:17.963646 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ff92fc16244b87b6d30cf38194182d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:17.966934 1583 raft_consensus.cc:3054] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:17.957311 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 60160.8us
I20250114 02:54:17.924561 2878 raft_consensus.cc:2798] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:17.961367 2710 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:17.950791 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:17.968585 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ff92fc16244b87b6d30cf38194182d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:17.968758 1655 raft_consensus.cc:1235] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250114 02:54:17.969100 1728 raft_consensus.cc:3054] T c4ff92fc16244b87b6d30cf38194182d P b89550620e1d455d887be72232bb5454 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.931739 2756 raft_consensus.cc:2798] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 02:54:17.931473 2843 raft_consensus.cc:2743] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 02:54:17.924793 2868 consensus_queue.cc:260] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.974215 2868 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:17.974494 3232 raft_consensus.cc:2743] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250114 02:54:17.975242 2916 consensus_queue.cc:1159] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" responder_term: 6 status { last_received { term: 6 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 6 index: 4 } }
I20250114 02:54:17.976256 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:17.976840 1582 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:17.986521 2878 raft_consensus.cc:695] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:17.987622 2878 consensus_queue.cc:237] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.988931 2710 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:17.991279 2710 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:17.966444 3237 raft_consensus.cc:491] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:17.992134 3237 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:17.991586 1728 raft_consensus.cc:2463] T c4ff92fc16244b87b6d30cf38194182d P b89550620e1d455d887be72232bb5454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:54:17.995106 1702 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:17.996007 3146 raft_consensus.cc:2798] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:17.996546 3146 raft_consensus.cc:695] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:17.997642 3146 consensus_queue.cc:237] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.011628 2971 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:18.011822 2868 consensus_queue.cc:1046] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:54:18.012774 3152 raft_consensus.cc:3049] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 02:54:18.013624 3152 raft_consensus.cc:738] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:18.014355 3152 consensus_queue.cc:260] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.015681 3152 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:17.961113 3140 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:17.890082 3072 consensus_queue.cc:1035] T 6bae187c925c49dd9bb36f34be387836 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
I20250114 02:54:18.017894 3092 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:18.019274 3153 raft_consensus.cc:2743] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 02:54:18.019907 3153 consensus_queue.cc:1035] T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20250114 02:54:18.020713 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 47992.3us
I20250114 02:54:18.023684 3235 consensus_queue.cc:260] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.024737 3235 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:18.019954 3154 consensus_queue.cc:1035] T 4a25a35a51714764a2d368283f31d19a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:18.031004 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:18.031738 1655 raft_consensus.cc:2463] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:18.037998 1655 raft_consensus.cc:1270] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 02:54:18.038674 1583 raft_consensus.cc:2463] T c4ff92fc16244b87b6d30cf38194182d P 62daa58e8ff64698bb46ea3fd5290cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 1.
I20250114 02:54:18.039577 1702 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:18.048151 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:18.048864 1586 raft_consensus.cc:2463] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 3.
I20250114 02:54:18.054425 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:18.051577 3140 raft_consensus.cc:695] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
W20250114 02:54:18.086269 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 121035us
I20250114 02:54:18.092267 2756 raft_consensus.cc:695] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
W20250114 02:54:18.092787 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 63870.3us
I20250114 02:54:18.119917 1628 leader_election.cc:304] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:18.022523 3153 consensus_queue.cc:1035] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20250114 02:54:18.120965 1580 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:18.059558 2710 raft_consensus.cc:2798] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 02:54:18.117745 1553 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:18.189410 3248 raft_consensus.cc:2798] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:18.190699 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:18.071624 3147 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:18.047757 3075 consensus_queue.cc:1035] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20250114 02:54:18.197290 1655 raft_consensus.cc:2388] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:18.198884 3248 raft_consensus.cc:695] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:18.200790 3152 consensus_queue.cc:1035] T 88a6663722394089be1417d1cf01653a P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:18.200884 3248 consensus_queue.cc:237] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.208129 3103 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:18.122282 2710 raft_consensus.cc:491] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:18.210523 3140 consensus_queue.cc:237] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:18.215767 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 98084.2us
I20250114 02:54:18.216892 1553 leader_election.cc:304] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:18.225769 1628 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:18.229818 1728 raft_consensus.cc:1270] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:18.229918 1702 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:18.233619 1579 raft_consensus.cc:1270] T 3a4107caddb14178ae1ca233127cc4a3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:18.212914 2710 raft_consensus.cc:3054] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:18.221997 3092 raft_consensus.cc:2743] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 02:54:18.257388 2756 consensus_queue.cc:237] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.241204 3153 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:18.268420 3153 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:18.268877 3153 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:18.227507 3259 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.274832 1654 raft_consensus.cc:1270] T f704f7a8203f4a5fabb81d95f3a5f38e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:18.283687 1653 raft_consensus.cc:1270] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 02:54:18.283874 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:18.284735 1728 raft_consensus.cc:2405] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20250114 02:54:18.285899 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 44813.9us
I20250114 02:54:18.289155 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:18.292207 1727 raft_consensus.cc:1270] T c818f667d1314fc68ee42a2fd1c5a627 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:18.295567 1735 raft_consensus.cc:1270] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 02:54:18.297981 1731 raft_consensus.cc:1270] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 02:54:18.298841 1732 raft_consensus.cc:1270] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20250114 02:54:18.299155 1732 pending_rounds.cc:77] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 3
I20250114 02:54:18.299343 1732 pending_rounds.cc:99] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20250114 02:54:18.299538 1732 raft_consensus.cc:2883] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:18.303210 1587 raft_consensus.cc:1270] T c818f667d1314fc68ee42a2fd1c5a627 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250114 02:54:18.304232 1628 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:18.286768 3009 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:18.341928 3140 raft_consensus.cc:2743] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 02:54:18.343477 3145 consensus_queue.cc:1035] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:18.343557 3140 consensus_queue.cc:1035] T 44b972e53ce14bd0825dc78f9c5c1a27 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:18.344753 3145 consensus_queue.cc:1035] T c818f667d1314fc68ee42a2fd1c5a627 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:18.283967 3260 consensus_queue.cc:1035] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20250114 02:54:18.285487 3259 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.307282 3232 consensus_queue.cc:1035] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20250114 02:54:18.335410 3131 consensus_queue.cc:1035] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20250114 02:54:18.376168 1734 raft_consensus.cc:1270] T 1fdaf3f35acd45469c23cec4d4112b11 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:18.348968 3009 raft_consensus.cc:695] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:18.291693 3271 consensus_queue.cc:1035] T 7828a26fd7794b8a9ae9737e193416d2 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
@ 0x7fbce3392e30 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
I20250114 02:54:18.369112 3150 consensus_queue.cc:1035] T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20250114 02:54:18.449208 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 52724us
I20250114 02:54:18.381685 3160 consensus_queue.cc:1035] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.059s
I20250114 02:54:18.388643 3235 consensus_queue.cc:1035] T f704f7a8203f4a5fabb81d95f3a5f38e P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20250114 02:54:18.391428 2968 consensus_queue.cc:1035] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
W20250114 02:54:18.445386 3277 log.cc:926] Time spent T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250114 02:54:18.433689 3273 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.469424 3273 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.465981 3259 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:18.471222 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:18.454062 3237 raft_consensus.cc:513] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.472731 2796 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.473023 3009 consensus_queue.cc:237] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.461503 3072 raft_consensus.cc:2743] T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:18.475814 3086 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:18.477910 3263 log.cc:926] Time spent T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.175s user 0.001s sys 0.000s
I20250114 02:54:18.502768 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
W20250114 02:54:18.518146 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 47274.3us
I20250114 02:54:18.522257 1579 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:18.523741 1628 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:18.525581 3090 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:18.526912 3090 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:18.527309 3090 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:18.521162 2796 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.530687 2868 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.531177 2868 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.531992 2796 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:18.532922 2868 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:18.524394 3086 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.534534 3090 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.535197 3086 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:18.537077 3090 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:18.540827 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:18.541524 1662 raft_consensus.cc:2463] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 7.
I20250114 02:54:18.542491 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:18.546075 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:18.546525 1584 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:18.548108 1586 raft_consensus.cc:1270] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:18.549602 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:18.550691 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:18.557896 1735 raft_consensus.cc:1270] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:18.565440 1584 raft_consensus.cc:2463] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 7.
I20250114 02:54:18.566277 1628 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:18.566823 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:18.571000 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:18.572803 1582 raft_consensus.cc:1270] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:18.585639 3227 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.586067 3227 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.587802 3227 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:18.588255 3227 consensus_queue.cc:1035] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 02:54:18.588537 1702 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:18.589591 1702 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:18.567592 1655 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:18.596426 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:18.597985 3215 raft_consensus.cc:2798] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 02:54:18.598362 3215 raft_consensus.cc:491] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:18.598697 3215 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Advancing to term 8
I20250114 02:54:18.575870 3249 raft_consensus.cc:2798] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 02:54:18.599469 3249 raft_consensus.cc:695] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:18.603037 1553 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters: b89550620e1d455d887be72232bb5454
I20250114 02:54:18.600306 3249 consensus_queue.cc:237] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.606314 3237 leader_election.cc:290] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:18.607227 3289 log.cc:926] Time spent T 2fccc76f52ed4204b3a8b03af5f65b1d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20250114 02:54:18.609164 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:18.609891 1655 raft_consensus.cc:1270] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:18.590497 3238 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.545092 3153 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.652905 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:18.654222 1582 raft_consensus.cc:2405] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250114 02:54:18.660172 1655 raft_consensus.cc:1270] T f83a9dac23e1497286938e63aa482439 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250114 02:54:18.670210 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 67250.7us
W20250114 02:54:18.672022 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 72546us
I20250114 02:54:18.672888 1657 raft_consensus.cc:1235] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7. Ops: []
I20250114 02:54:18.597154 3013 consensus_queue.cc:1035] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.065s
I20250114 02:54:18.601297 2923 consensus_queue.cc:1035] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
W20250114 02:54:18.704362 3288 log.cc:926] Time spent T 4a25a35a51714764a2d368283f31d19a P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.181s user 0.001s sys 0.000s
I20250114 02:54:18.703989 2926 raft_consensus.cc:2798] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:18.728715 2926 raft_consensus.cc:491] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:18.729103 2926 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
W20250114 02:54:18.730999 3293 log.cc:926] Time spent T 6bae187c925c49dd9bb36f34be387836 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250114 02:54:18.721877 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:18.667145 3155 consensus_queue.cc:1035] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20250114 02:54:18.732816 1728 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:18.713946 2710 raft_consensus.cc:513] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.781162 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a8d6305b54a66ab67e039cfaaee26" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:18.783063 1728 raft_consensus.cc:2463] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:18.704216 3075 consensus_queue.cc:1035] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
W20250114 02:54:18.860248 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 188165us
I20250114 02:54:18.861896 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:18.783999 3277 log.cc:926] Time spent T 2fccc76f52ed4204b3a8b03af5f65b1d P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20250114 02:54:18.863518 3286 log.cc:926] Time spent T 7828a26fd7794b8a9ae9737e193416d2 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.346s user 0.001s sys 0.000s
I20250114 02:54:18.664206 3238 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.844870 3273 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:18.894713 1656 raft_consensus.cc:2405] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250114 02:54:18.904522 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:18.912742 3238 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:18.913949 2710 leader_election.cc:290] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:18.918059 3297 log.cc:926] Time spent T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250114 02:54:18.928660 3238 consensus_queue.cc:1035] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:18.902937 3153 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:18.947394 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:18.948066 1583 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:18.950618 1628 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
@ 0x7fbce3370a32 testing::Test::Run() at ??:0
I20250114 02:54:18.957185 3305 raft_consensus.cc:2798] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:18.957538 3305 raft_consensus.cc:491] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:18.957859 3305 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:18.959663 2926 raft_consensus.cc:513] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.962319 2926 leader_election.cc:290] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:18.966799 1585 raft_consensus.cc:1270] T 1fdaf3f35acd45469c23cec4d4112b11 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:18.976531 3259 consensus_queue.cc:1035] T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:18.977365 3080 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:18.977840 3080 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.982592 2710 raft_consensus.cc:491] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.983162 2710 raft_consensus.cc:513] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.983825 3176 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:18.983951 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:18.984536 1579 raft_consensus.cc:3054] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Advancing to term 4
I20250114 02:54:18.984385 3176 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.985395 2710 leader_election.cc:290] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:18.985975 2710 raft_consensus.cc:491] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.986114 3176 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:18.986456 2710 raft_consensus.cc:513] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.993973 3215 raft_consensus.cc:513] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:18.998612 1628 leader_election.cc:304] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:19.002452 1579 raft_consensus.cc:2463] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:19.015270 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.015929 1736 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:19.021968 2710 leader_election.cc:290] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:19.031898 3305 raft_consensus.cc:513] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.020321 3238 raft_consensus.cc:2798] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election won for term 5
W20250114 02:54:19.040608 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 177569us
I20250114 02:54:19.045222 1702 leader_election.cc:304] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:19.044067 2878 raft_consensus.cc:2743] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:19.081471 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:19.091624 1657 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:19.057341 3238 raft_consensus.cc:695] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:19.110925 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:19.111517 1587 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:19.101435 3114 raft_consensus.cc:2798] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 02:54:19.102687 3140 raft_consensus.cc:491] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:19.123227 1657 raft_consensus.cc:2463] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:19.105207 3176 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.135071 1553 leader_election.cc:304] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:19.142607 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:19.148761 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.149372 1733 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:19.150329 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8434d11346144778e385ac18d417859" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:19.150765 1733 raft_consensus.cc:3054] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:19.155298 1583 raft_consensus.cc:1270] T 903edcc3b2294723af07e77c238bd3c8 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 02:54:19.138423 3080 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:19.134380 3114 raft_consensus.cc:695] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:19.145613 3270 raft_consensus.cc:2798] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:19.145274 3305 leader_election.cc:290] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.158321 3129 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.134330 3140 raft_consensus.cc:513] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.183133 1733 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:19.137229 3176 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.192417 3270 raft_consensus.cc:695] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
W20250114 02:54:19.203708 3318 log.cc:926] Time spent T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20250114 02:54:19.182401 3233 consensus_queue.cc:1035] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20250114 02:54:19.159217 3238 consensus_queue.cc:237] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.206283 3114 consensus_queue.cc:237] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.215922 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024e8460e81348268cc6fd408ece2a86" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:19.216867 1660 raft_consensus.cc:3054] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Advancing to term 4
W20250114 02:54:19.221035 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 48955.4us
I20250114 02:54:19.188359 3129 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.201910 3270 consensus_queue.cc:237] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.241439 1660 raft_consensus.cc:2463] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 4.
I20250114 02:54:19.254734 1587 raft_consensus.cc:2463] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:19.282697 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.283381 1657 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 6.
I20250114 02:54:19.287159 3215 leader_election.cc:290] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
W20250114 02:54:19.289089 3285 log.cc:926] Time spent T 44b972e53ce14bd0825dc78f9c5c1a27 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.763s user 0.013s sys 0.000s
I20250114 02:54:19.292093 1553 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:19.293813 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34928 (request call id 835) took 1015 ms. Trace:
I20250114 02:54:19.275123 3140 leader_election.cc:290] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.281286 3129 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.291203 2868 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.296015 3232 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:19.368187 2796 raft_consensus.cc:491] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:19.296283 1581 rpcz_store.cc:276] 0114 02:54:18.278048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:18.278183 (+ 135us) service_pool.cc:225] Handling call
0114 02:54:18.293855 (+ 15672us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:18.313301 (+ 19446us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:18.313327 (+ 26us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:18.331845 (+ 18518us) log.cc:844] Serialized 26 byte log entry
0114 02:54:18.418262 (+ 86417us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:18.418312 (+ 50us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:18.418458 (+ 146us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:19.288554 (+870096us) spinlock_profiling.cc:243] Waited 3.35 ms on lock 0x7b440002494c. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce30ad7c5 00007fbce30ae1e2 00007fbce5d63da1 00007fbce5d5f68a 00007fbce5d534ae 00007fbce5d527f6 00007fbce5d52418 00007fbce5d5bdf1 00007fbce5d5bda9 00007fbce5d5bd39
0114 02:54:19.290912 (+ 2358us) raft_consensus.cc:1680] finished
0114 02:54:19.290932 (+ 20us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:19.293751 (+ 2819us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1473,"raft.queue_time_us":6202,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":16751,"spinlock_wait_cycles":7704320,"thread_start_us":65043,"threads_started":1,"wal-append.queue_time_us":132804}
I20250114 02:54:19.371824 2923 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:19.372349 3313 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:19.372920 3086 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.373271 3273 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:19.303681 3176 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:19.401844 3327 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.405081 3233 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.404879 3237 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.402876 2926 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:19.428429 3273 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.406852 3232 raft_consensus.cc:491] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:19.424077 3129 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:19.522980 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 277568us
I20250114 02:54:19.406347 2868 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.531808 1628 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:19.425848 2796 raft_consensus.cc:513] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.503908 3176 raft_consensus.cc:491] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:19.503523 3232 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:19.426688 2923 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.429185 3086 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.429523 3313 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.524446 3075 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.520386 3129 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:19.560704 1702 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.011s sys 0.010s
W20250114 02:54:19.561085 1702 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.515s user 0.011s sys 0.010s
I20250114 02:54:19.560927 3232 raft_consensus.cc:513] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:19.561354 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 516224us
I20250114 02:54:19.561909 3129 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.548385 3310 raft_consensus.cc:2743] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 02:54:19.562980 3232 leader_election.cc:290] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.546281 2796 leader_election.cc:290] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:19.546420 2868 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:19.501869 3237 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.549409 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.539625 3176 raft_consensus.cc:513] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.568578 1656 raft_consensus.cc:2388] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 7: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:19.530619 3215 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.569322 3215 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.569967 3176 leader_election.cc:290] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:19.546444 2923 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.571094 3313 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.571805 3237 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.553648 3273 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.465138 3233 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.504981 2926 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.574277 3233 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.571136 3215 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:19.557379 3086 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.462059 3327 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.576431 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.585083 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f704f7a8203f4a5fabb81d95f3a5f38e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:19.559351 3075 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
@ 0x7fbce3371d49 testing::TestInfo::Run() at ??:0
I20250114 02:54:19.577788 1553 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:19.578281 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:19.588266 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:19.588762 1579 raft_consensus.cc:2363] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:19.588794 1587 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:19.579943 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:19.582055 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4107caddb14178ae1ca233127cc4a3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.586311 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.590029 1661 raft_consensus.cc:2388] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:19.592080 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.592437 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.592644 1656 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:19.594003 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:19.593966 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:19.594923 1587 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:19.601327 1660 raft_consensus.cc:1235] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5. Ops: []
I20250114 02:54:19.604063 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.597608 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.605741 1661 raft_consensus.cc:2388] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:19.617347 1702 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:19.617218 1581 raft_consensus.cc:1235] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7. Ops: []
I20250114 02:54:19.619263 1656 raft_consensus.cc:1235] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7. Ops: []
I20250114 02:54:19.596726 3154 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.619891 3154 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.620785 1585 raft_consensus.cc:1270] T f83a9dac23e1497286938e63aa482439 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:19.621922 3154 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.626087 3327 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.629315 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.630131 1659 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:19.622503 1655 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 6.
I20250114 02:54:19.635735 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:19.641017 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b78b6bf8a94ca2b4d5f71aaad16e83" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:19.599298 3270 raft_consensus.cc:2758] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250114 02:54:19.646732 1727 raft_consensus.cc:1235] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 02:54:19.647801 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.648640 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
W20250114 02:54:19.648960 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 117281us
I20250114 02:54:19.591449 3215 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:19.649199 1735 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 7.
I20250114 02:54:19.649013 1736 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:19.649961 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.651646 1661 raft_consensus.cc:2388] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 7: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:19.650061 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.652226 1728 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 6.
I20250114 02:54:19.653450 1553 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:19.655311 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.655403 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:19.655793 1728 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 4.
I20250114 02:54:19.656085 1659 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:19.609344 2926 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.658844 1732 raft_consensus.cc:1235] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250114 02:54:19.660948 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:19.661517 1585 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:19.643107 3114 raft_consensus.cc:2743] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:19.665995 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9222bb8af9ab4f35a3fa735c302f11ac" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:19.666954 1728 raft_consensus.cc:3054] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 7
I20250114 02:54:19.670327 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.671344 2923 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:19.675604 2923 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.677462 2923 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.659736 3215 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.678712 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.606053 2868 consensus_queue.cc:1046] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20250114 02:54:19.679942 2868 raft_consensus.cc:3049] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 02:54:19.679883 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edafd45551ec43f9a5b750c1e54c62f6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.680333 2868 raft_consensus.cc:738] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:19.679306 1654 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 7.
I20250114 02:54:19.681037 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.684566 1661 raft_consensus.cc:1235] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 02:54:19.681599 1662 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:19.660583 3313 raft_consensus.cc:2798] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:19.684059 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0a423b0ba64289acb34e5c4d31aad0" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:19.689183 1728 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 7.
I20250114 02:54:19.666246 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.690441 1628 leader_election.cc:304] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:19.702224 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:19.703090 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.707221 1735 raft_consensus.cc:1270] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 02:54:19.711947 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:19.712869 1579 raft_consensus.cc:2405] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250114 02:54:19.712752 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:19.713557 1587 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:19.713918 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:19.714732 1583 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:19.711103 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f2bc8299cf4834bebd9aa5ae95a56e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.718004 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:19.679323 3215 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:19.724963 1659 raft_consensus.cc:2463] T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 7.
I20250114 02:54:19.764801 1587 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 4.
I20250114 02:54:19.703867 1653 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 5.
I20250114 02:54:19.725889 1734 raft_consensus.cc:1270] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:19.726799 1729 raft_consensus.cc:3049] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 02:54:19.765483 1729 raft_consensus.cc:738] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:19.717607 3233 raft_consensus.cc:2798] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:19.731384 2923 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.689121 3313 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:19.766234 2923 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.767097 3233 raft_consensus.cc:695] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:19.768321 2923 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.746843 3075 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:19.681066 2868 consensus_queue.cc:260] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.762137 1655 raft_consensus.cc:1270] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:19.707474 1727 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 6.
I20250114 02:54:19.769562 1553 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:19.790937 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.793727 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.788712 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.791736 1736 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 3.
I20250114 02:54:19.796684 1553 leader_election.cc:304] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:19.797785 1553 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:19.799468 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.800561 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.801493 1731 raft_consensus.cc:2388] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 5: Already voted for candidate b89550620e1d455d887be72232bb5454 in this term.
I20250114 02:54:19.802501 1731 raft_consensus.cc:3049] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:19.807222 1731 raft_consensus.cc:738] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:19.800225 1736 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 7.
I20250114 02:54:19.824810 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.825531 1736 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:19.830154 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.832468 1702 leader_election.cc:400] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:19.832899 1702 leader_election.cc:403] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250114 02:54:19.831596 1736 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 6.
I20250114 02:54:19.796841 1729 consensus_queue.cc:260] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.835743 1702 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:19.766248 3313 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:19.837556 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:19.808347 1731 consensus_queue.cc:260] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.840644 1702 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:19.834486 1729 raft_consensus.cc:3054] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:19.849995 1734 raft_consensus.cc:2405] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250114 02:54:19.767992 3233 consensus_queue.cc:237] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.825620 2923 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:19.839248 3115 raft_consensus.cc:2798] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election won for term 7
I20250114 02:54:19.837245 3338 raft_consensus.cc:2798] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:19.851483 2868 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:19.849330 2926 raft_consensus.cc:491] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:19.857244 3114 raft_consensus.cc:2798] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:19.875408 2923 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:19.837412 3270 consensus_queue.cc:1035] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20250114 02:54:19.851217 3176 raft_consensus.cc:2743] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:19.900508 3338 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:19.914670 3115 raft_consensus.cc:695] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:19.936142 3115 consensus_queue.cc:237] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.946580 3086 consensus_queue.cc:1046] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 02:54:19.947548 3086 consensus_queue.cc:1225] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer 1e5870f8269b49cf81dc39f1a5c8f5c1 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250114 02:54:19.948074 3086 raft_consensus.cc:3049] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Stepping down as leader of term 4
I20250114 02:54:19.948432 3086 raft_consensus.cc:738] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:19.949298 3086 consensus_queue.cc:260] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.913720 2926 raft_consensus.cc:513] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.951625 3232 raft_consensus.cc:491] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:19.952142 3232 raft_consensus.cc:513] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.953112 2926 leader_election.cc:290] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.953572 3302 raft_consensus.cc:491] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:19.953881 3232 leader_election.cc:290] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:19.954077 3302 raft_consensus.cc:513] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:19.956787 3302 leader_election.cc:290] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
W20250114 02:54:19.958817 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:54:19.915323 2923 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:19.987834 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.989372 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.991334 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:19.910581 3114 raft_consensus.cc:491] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:19.995596 3114 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:19.927136 3338 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
W20250114 02:54:20.015057 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 174490us
I20250114 02:54:20.016582 3013 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 02:54:20.017021 3013 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:20.017452 3013 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Advancing to term 8
I20250114 02:54:20.020298 3147 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:20.020910 3147 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.025832 2878 consensus_queue.cc:1035] T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20250114 02:54:20.028322 3147 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:20.038758 1731 raft_consensus.cc:3054] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Advancing to term 5
@ 0x7fbce3372d25 testing::TestSuite::Run() at ??:0
I20250114 02:54:20.060410 3086 raft_consensus.cc:3054] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:20.064153 3160 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.064615 3160 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.068210 3160 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.061511 3302 raft_consensus.cc:491] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:20.079560 3302 raft_consensus.cc:513] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.089687 3270 raft_consensus.cc:491] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.090288 3270 raft_consensus.cc:513] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.100713 3072 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:20.101305 3072 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:20.156078 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 91868.9us
I20250114 02:54:20.157073 3302 leader_election.cc:290] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.166160 3270 leader_election.cc:290] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.166860 1628 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:20.168444 3331 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:20.168810 3331 raft_consensus.cc:491] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:20.173022 1731 raft_consensus.cc:1270] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250114 02:54:20.174314 1731 pending_rounds.cc:77] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 2
I20250114 02:54:20.175272 1731 pending_rounds.cc:99] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250114 02:54:20.175570 1628 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
W20250114 02:54:20.175668 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 149256us
I20250114 02:54:20.176486 1731 raft_consensus.cc:2883] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:20.176623 1702 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:20.180444 1729 raft_consensus.cc:1270] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20250114 02:54:20.180861 1729 pending_rounds.cc:77] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454: Aborting all ops after (but not including) 4
I20250114 02:54:20.181121 1729 pending_rounds.cc:99] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20250114 02:54:20.181388 1729 raft_consensus.cc:2883] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:20.183933 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:20.197880 1628 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:20.169165 3331 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:20.246347 3345 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.247047 1628 leader_election.cc:304] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:20.218375 3324 raft_consensus.cc:2798] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:20.248399 3300 consensus_queue.cc:1035] T 903edcc3b2294723af07e77c238bd3c8 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 02:54:20.248622 2971 raft_consensus.cc:2743] T c1f2bc8299cf4834bebd9aa5ae95a56e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:20.248847 3324 raft_consensus.cc:491] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:20.249265 3324 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:20.219286 2796 raft_consensus.cc:491] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:20.254837 2796 raft_consensus.cc:513] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.256855 2796 leader_election.cc:290] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:20.219862 3147 raft_consensus.cc:491] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.223415 1581 raft_consensus.cc:1270] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 02:54:20.198890 3153 raft_consensus.cc:2798] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 02:54:20.257522 3147 raft_consensus.cc:513] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.257968 3153 raft_consensus.cc:491] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:20.258343 3153 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:20.259464 3147 leader_election.cc:290] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:20.224252 1731 raft_consensus.cc:1270] T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:20.262231 2885 consensus_queue.cc:1035] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 02:54:20.263746 1628 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:20.224769 3072 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.224336 3313 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.225239 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:20.266669 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:20.267313 1656 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:20.230440 3013 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:20.231359 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 54837.6us
I20250114 02:54:20.271548 1580 raft_consensus.cc:1270] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250114 02:54:20.276961 1580 pending_rounds.cc:77] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 2
I20250114 02:54:20.233543 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:20.277192 1580 pending_rounds.cc:99] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250114 02:54:20.277613 1736 raft_consensus.cc:2405] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250114 02:54:20.235832 3227 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:20.287071 1654 raft_consensus.cc:1270] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:20.283372 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:20.246830 3345 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.267827 1482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39732 (request call id 93) took 5806 ms. Trace:
I20250114 02:54:20.164875 2923 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.276559 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:20.277416 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:20.236938 1578 raft_consensus.cc:1270] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20250114 02:54:20.304373 1578 pending_rounds.cc:77] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting all ops after (but not including) 3
I20250114 02:54:20.304672 1578 pending_rounds.cc:99] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250114 02:54:20.304929 1578 raft_consensus.cc:2883] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:20.278052 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:20.232007 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:20.277442 1580 raft_consensus.cc:2883] T 460a8d6305b54a66ab67e039cfaaee26 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:20.288666 1656 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 6.
I20250114 02:54:20.289683 1553 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:20.286375 1483 catalog_manager.cc:5526] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 1 to 5, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 5 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
W20250114 02:54:20.297808 1730 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.005s sys 0.017s
I20250114 02:54:20.300563 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:20.309026 1581 raft_consensus.cc:2405] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250114 02:54:20.300920 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:20.302860 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:20.310978 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccc76f52ed4204b3a8b03af5f65b1d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:20.269711 3326 raft_consensus.cc:2743] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 02:54:20.295565 1482 rpcz_store.cc:276] 0114 02:54:14.514922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.524450 (+ 9528us) service_pool.cc:225] Handling call
0114 02:54:20.267759 (+5743309us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:18.205185 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:18.205674 (+ 489us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:18.205703 (+ 29us) write_op.cc:435] Acquired schema lock
0114 02:54:18.205716 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:18.207125 (+ 1409us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:18.207181 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:18.207208 (+ 27us) tablet.cc:682] Acquiring locks for 19 operations
0114 02:54:18.209835 (+ 2627us) tablet.cc:698] Row locks acquired
0114 02:54:18.209851 (+ 16us) write_op.cc:260] PREPARE: finished
0114 02:54:18.209975 (+ 124us) write_op.cc:270] Start()
0114 02:54:18.210077 (+ 102us) write_op.cc:276] Timestamp: P: 1736823258209934 usec, L: 0
0114 02:54:18.210094 (+ 17us) op_driver.cc:348] REPLICATION: starting
0114 02:54:18.233228 (+ 23134us) log.cc:844] Serialized 7464 byte log entry
0114 02:54:18.721303 (+488075us) op_driver.cc:464] REPLICATION: finished
0114 02:54:18.866770 (+145467us) spinlock_profiling.cc:243] Waited 4.56 ms on lock 0x7b54002b02d4. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce605a6d1 00007fbce60d1198 00007fbce9ae83b1 00007fbce9ae7b88 00007fbce9aec821 00007fbce9aec7d9 00007fbce9aec769 00007fbce9aec731 00007fbce9aeb95d 00007fbcec1d9234 00007fbcec1d9069
0114 02:54:18.956879 (+ 90109us) write_op.cc:301] APPLY: starting
0114 02:54:18.956945 (+ 66us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:18.957304 (+ 359us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:18.957327 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:18.959204 (+ 1877us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:18.959453 (+ 249us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=19
0114 02:54:18.959497 (+ 44us) write_op.cc:312] APPLY: finished
0114 02:54:18.960740 (+ 1243us) log.cc:844] Serialized 131 byte log entry
0114 02:54:18.962351 (+ 1611us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:18.962505 (+ 154us) write_op.cc:454] Released schema lock
0114 02:54:18.962569 (+ 64us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:19.012624 (+ 50055us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:19.025671 (+ 13047us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3136,"reported_tablets":54,"child_traces":[["op",{"apply.queue_time_us":230095,"apply.run_cpu_time_us":3685,"apply.run_wall_time_us":16518,"num_ops":19,"prepare.queue_time_us":240646,"prepare.run_cpu_time_us":41906,"prepare.run_wall_time_us":666507,"raft.queue_time_us":275362,"raft.run_cpu_time_us":37119,"raft.run_wall_time_us":225386,"replication_time_us":466231,"spinlock_wait_cycles":10482432,"thread_start_us":474548,"threads_started":4,"wal-append.queue_time_us":146433,"wal-append.run_cpu_time_us":17584,"wal-append.run_wall_time_us":1575518}]]}
I20250114 02:54:20.308799 1483 catalog_manager.cc:5526] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 2 to 6, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 6 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:20.318156 1655 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:20.319329 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:20.319433 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:20.320209 1582 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 6.
I20250114 02:54:20.287144 3227 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 02:54:20.313339 1730 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s user 0.005s sys 0.017s
I20250114 02:54:20.321552 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:20.320806 3083 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:20.323431 3083 raft_consensus.cc:491] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:20.323765 3083 raft_consensus.cc:3054] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:20.323835 3013 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:20.326733 3154 consensus_queue.cc:1035] T c4ff92fc16244b87b6d30cf38194182d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:20.327898 3154 consensus_queue.cc:1035] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:20.329639 3338 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.330778 3153 raft_consensus.cc:513] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.331681 3338 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.332548 3153 leader_election.cc:290] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.303612 3313 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.334724 3331 raft_consensus.cc:513] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.336072 1662 raft_consensus.cc:2363] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5.
I20250114 02:54:20.336587 3331 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.338716 3345 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.340471 2923 leader_election.cc:290] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.338449 1658 raft_consensus.cc:2363] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:20.309604 2926 raft_consensus.cc:491] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:20.340946 3324 raft_consensus.cc:513] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.341594 2926 raft_consensus.cc:513] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.343343 2926 leader_election.cc:290] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.344957 3083 raft_consensus.cc:513] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.345291 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:20.347725 1578 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 5.
I20250114 02:54:20.349660 3083 leader_election.cc:290] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.350859 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:20.351491 1735 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Advancing to term 5
I20250114 02:54:20.346743 3227 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:20.347254 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 985a08c846764478aaeb97aa5c283bf9 with delete_type TABLET_DATA_DELETED (Replaced by c4ff92fc16244b87b6d30cf38194182d at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:48630
I20250114 02:54:20.352209 1581 raft_consensus.cc:2388] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 4: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:20.314541 1553 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
W20250114 02:54:20.359290 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 44833.9us
I20250114 02:54:20.356392 1483 catalog_manager.cc:5526] T 0b78459d23a744819d55e5f50de8174f P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 3 to 5, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 5 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:20.362988 1581 raft_consensus.cc:1270] T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250114 02:54:20.366685 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:20.370137 1727 raft_consensus.cc:2405] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 02:54:20.364666 1483 catalog_manager.cc:5526] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 2 to 3, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 3 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:20.376590 1483 catalog_manager.cc:5526] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 3 to 4, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 4 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:20.377411 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:20.378029 1728 raft_consensus.cc:3054] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 7
I20250114 02:54:20.374122 1735 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:20.379287 1581 raft_consensus.cc:1235] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 LEADER]: Rejecting Update request from peer b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250114 02:54:20.387141 1553 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:20.390003 1702 leader_election.cc:304] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:20.375936 3083 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:20.391113 3083 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:20.391485 3083 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:20.299823 2885 consensus_queue.cc:1035] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20250114 02:54:20.326272 3114 raft_consensus.cc:513] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.401530 3300 consensus_queue.cc:1035] T f37e0491ebc34f29b68a6425e22a3a2d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 02:54:20.390355 1628 leader_election.cc:304] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:20.406817 1702 leader_election.cc:304] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:20.411262 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:20.411974 1733 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:20.407550 1553 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:20.399905 3324 leader_election.cc:290] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.419349 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:20.428299 1580 raft_consensus.cc:2363] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:20.428882 1733 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
I20250114 02:54:20.430044 1628 leader_election.cc:304] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:20.381639 3086 consensus_queue.cc:1046] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20250114 02:54:20.409489 2926 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 02:54:20.435348 1702 leader_election.cc:304] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:20.443543 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464b2086919241de965a1ac17809451c" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:20.444584 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fec1da7d2f8404bb83e600bc99eb944" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:20.445082 1584 raft_consensus.cc:3054] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Advancing to term 5
I20250114 02:54:20.414798 3176 raft_consensus.cc:2743] T 3a4107caddb14178ae1ca233127cc4a3 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 02:54:20.412603 3145 raft_consensus.cc:2798] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 02:54:20.453716 1628 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:20.441608 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903edcc3b2294723af07e77c238bd3c8" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:20.441938 1728 raft_consensus.cc:2463] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 7.
I20250114 02:54:20.453775 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:20.457161 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:20.457893 1578 raft_consensus.cc:3054] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:20.459820 1584 raft_consensus.cc:2463] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 5.
I20250114 02:54:20.463651 1553 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:20.440407 1483 catalog_manager.cc:5526] T 4876e2bdbafa4d8684adc1348f1aeb80 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 2 to 6, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 6 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:20.465044 1628 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:20.476636 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:20.476784 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:20.477635 1580 raft_consensus.cc:2388] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in current term 7: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:20.452559 2926 raft_consensus.cc:491] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:20.482834 2926 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:20.464171 3178 consensus_queue.cc:1035] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.020s
I20250114 02:54:20.454978 2923 raft_consensus.cc:491] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:20.484977 3083 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.485730 2923 raft_consensus.cc:513] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.486653 3313 consensus_queue.cc:1035] T 29b78b6bf8a94ca2b4d5f71aaad16e83 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:20.486968 1586 raft_consensus.cc:2405] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250114 02:54:20.487768 2923 leader_election.cc:290] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.454775 3310 raft_consensus.cc:2798] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:20.488467 3149 consensus_queue.cc:1035] T 460a8d6305b54a66ab67e039cfaaee26 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:20.460220 3114 leader_election.cc:290] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:20.489064 3083 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.491218 3340 raft_consensus.cc:2798] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 02:54:20.491741 3340 raft_consensus.cc:695] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:20.492681 3340 consensus_queue.cc:237] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.492424 1483 catalog_manager.cc:5526] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 3 to 5, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 5 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:20.494235 1483 catalog_manager.cc:5526] T b4aa8b25b08141db9f0576ce3fa34d3c P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:20.496073 1578 raft_consensus.cc:2463] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in term 6.
@ 0x7fbce3387815 testing::internal::UnitTestImpl::RunAllTests() at ??:0
I20250114 02:54:20.444928 3072 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:20.464799 3215 raft_consensus.cc:2743] T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:20.508981 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:20.509403 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:20.504892 3072 raft_consensus.cc:491] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election (detected failure of leader b89550620e1d455d887be72232bb5454)
W20250114 02:54:20.510370 3350 log.cc:926] Time spent T 1fdaf3f35acd45469c23cec4d4112b11 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20250114 02:54:20.500869 3232 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:20.463654 3326 raft_consensus.cc:2743] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 02:54:20.514824 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:20.501055 3145 raft_consensus.cc:695] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:20.500504 3310 raft_consensus.cc:695] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 LEADER]: Becoming Leader. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:20.520864 1658 raft_consensus.cc:2388] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 7: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:20.517130 3232 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:20.517710 3072 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:20.537492 1553 leader_election.cc:304] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd; no voters:
I20250114 02:54:20.523115 3232 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:20.555696 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:20.495875 1483 catalog_manager.cc:5526] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 4 to 6, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 6 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:20.523995 3228 raft_consensus.cc:2743] T edafd45551ec43f9a5b750c1e54c62f6 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 02:54:20.574230 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:20.574683 1734 raft_consensus.cc:1270] T f8434d11346144778e385ac18d417859 P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250114 02:54:20.574976 1659 raft_consensus.cc:2388] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:20.562855 1483 catalog_manager.cc:5526] T 44b972e53ce14bd0825dc78f9c5c1a27 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 3 to 4, leader changed from 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 4 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:20.518868 3145 consensus_queue.cc:237] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.554616 2923 raft_consensus.cc:2798] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election won for term 6
I20250114 02:54:20.551611 2868 raft_consensus.cc:491] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:20.599552 1628 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:20.584656 1483 catalog_manager.cc:5526] T a26544b19eeb4116a06caaba1da0e9d2 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:20.611943 1483 catalog_manager.cc:5526] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 2 to 4, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 4 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:20.621654 1483 catalog_manager.cc:5526] T 01ce953d409541bb85818e3dcf36541b P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 3 to 5, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 5 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:20.602418 2923 raft_consensus.cc:695] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:20.610036 3345 raft_consensus.cc:2743] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 02:54:20.602748 2868 raft_consensus.cc:513] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:20.664367 3357 log.cc:926] Time spent T 3a4107caddb14178ae1ca233127cc4a3 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250114 02:54:20.609822 3233 consensus_queue.cc:1035] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20250114 02:54:20.677423 2923 consensus_queue.cc:237] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.679949 2868 leader_election.cc:290] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:20.696844 1483 catalog_manager.cc:5526] T f37e0491ebc34f29b68a6425e22a3a2d P b89550620e1d455d887be72232bb5454 reported cstate change: term changed from 4 to 5, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 5 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:20.593061 3310 consensus_queue.cc:237] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:20.703442 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 76410.9us
W20250114 02:54:20.744923 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 107074us
I20250114 02:54:20.776486 2926 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:20.759379 3347 log.cc:926] Time spent T f704f7a8203f4a5fabb81d95f3a5f38e P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.321s user 0.004s sys 0.000s
W20250114 02:54:20.788401 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 353162us
I20250114 02:54:20.790761 1702 leader_election.cc:304] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:20.791981 2796 raft_consensus.cc:2743] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250114 02:54:20.800302 1702 leader_election.cc:400] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5.
I20250114 02:54:20.800644 1702 leader_election.cc:403] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250114 02:54:20.801345 1702 leader_election.cc:400] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:20.801602 1702 leader_election.cc:403] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250114 02:54:20.803488 1702 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:20.808414 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d77d39f9a4b4a57ae17db8e50237cdb" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:20.808877 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3f8af715b4ed8a97965a52b089a21" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:20.809509 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:20.809767 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37e0491ebc34f29b68a6425e22a3a2d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:20.810190 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1c92618e4c44119a063bfdaa75fc3e" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:20.810910 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6c0814b8241c99bdeb347d765f929" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:20.812088 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26544b19eeb4116a06caaba1da0e9d2" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:20.813092 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07160adeb64840179d1c1529d86c94ed" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:20.813676 1733 raft_consensus.cc:2363] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6.
I20250114 02:54:20.816092 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b73d9b5d01245f8be76a02efe7e5eca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:20.816565 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:20.817131 1733 raft_consensus.cc:3054] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Advancing to term 6
I20250114 02:54:20.819378 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4876e2bdbafa4d8684adc1348f1aeb80" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:20.821523 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e00ae680c2d4c5cbb5154723228286f" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:20.805223 3009 raft_consensus.cc:2743] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 4. Current term is 5.
I20250114 02:54:20.822758 3176 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:20.823127 3176 raft_consensus.cc:491] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:20.823455 3176 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:20.823616 2926 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.828027 1733 raft_consensus.cc:2463] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 6.
I20250114 02:54:20.829684 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36452 (request call id 667) took 1175 ms. Trace:
I20250114 02:54:20.830174 1730 rpcz_store.cc:276] 0114 02:54:19.653922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.654052 (+ 130us) service_pool.cc:225] Handling call
0114 02:54:19.654371 (+ 319us) raft_consensus.cc:1466] Updating replica for 1 ops
0114 02:54:19.659356 (+ 4985us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 02:54:19.659385 (+ 29us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0114 02:54:19.670868 (+ 11483us) log.cc:844] Serialized 26 byte log entry
0114 02:54:19.751235 (+ 80367us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b1800290500. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce317e565 00007fbcebd907f9 00007fbce319ba6b 00007fbce3199e6f 00007fbce319bfd2 00007fbce5d51e4b 00007fbce5d57913 00007fbce5d57cc1 00007fbce60a945f
0114 02:54:20.365620 (+614385us) raft_consensus.cc:1640] Marking committed up to 2
0114 02:54:20.403522 (+ 37902us) log.cc:844] Serialized 14 byte log entry
0114 02:54:20.451549 (+ 48027us) spinlock_profiling.cc:243] Waited 1.53 ms on lock 0x7b1800290500. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce311403a 00007fbcec5d0be9 00007fbce5d6e820 00007fbce5d61e93 00007fbce5d578da 00007fbce5d57e2b 00007fbce60df97a 00007fbce60ef00f 00007fbce60eef88
0114 02:54:20.459452 (+ 7903us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 02:54:20.459551 (+ 99us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 02:54:20.663807 (+204256us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b1800290500. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbcebe16931 00007fbcebe23831 00007fbce319c914 00007fbce319fcc1 00007fbce319fc79 00007fbce319fc09 00007fbce319fbd1 00007fbce319eecd 00007fbcec1d9234 00007fbcec1d9069 00007fbce317efe6
0114 02:54:20.690338 (+ 26531us) spinlock_profiling.cc:243] Waited 9.29 ms on lock 0x7b440002494c. stack: 00007fbce3169f63 00007fbce3169e60 00007fbce2abd584 00007fbcec5b1ff4 00007fbcec5b1f89 00007fbce2fdf911 00007fbce30ad7c5 00007fbce30ae1e2 00007fbce5d63da1 00007fbce5d5f68a 00007fbce5d534ae 00007fbce5d527f6 00007fbce5d52418 00007fbce5d5bdf1 00007fbce5d5bda9 00007fbce5d5bd39
0114 02:54:20.829432 (+139094us) raft_consensus.cc:1680] finished
0114 02:54:20.829461 (+ 29us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 02:54:20.829618 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29034,"prepare.queue_time_us":38040,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":98,"raft.queue_time_us":12809,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":24398,"spinlock_wait_cycles":96604160,"thread_start_us":570786,"threads_started":1,"wal-append.queue_time_us":723728}
I20250114 02:54:20.807466 2916 raft_consensus.cc:2743] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:20.834832 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:20.834750 3072 raft_consensus.cc:513] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.835641 1653 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Advancing to term 8
I20250114 02:54:20.837601 3072 leader_election.cc:290] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:20.839205 1730 raft_consensus.cc:1270] T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 02:54:20.829701 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:20.844072 1728 raft_consensus.cc:2388] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 7: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:20.846930 1735 raft_consensus.cc:2463] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62daa58e8ff64698bb46ea3fd5290cdd in term 7.
I20250114 02:54:20.792596 2796 consensus_queue.cc:1035] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20250114 02:54:20.857842 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
W20250114 02:54:20.859495 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 74002.2us
I20250114 02:54:20.861573 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:20.862413 1657 raft_consensus.cc:2363] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 5. Current term is 6.
W20250114 02:54:20.862869 3365 log.cc:926] Time spent T 460a8d6305b54a66ab67e039cfaaee26 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20250114 02:54:20.871155 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:20.880002 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:20.880579 1579 raft_consensus.cc:3054] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Advancing to term 8
I20250114 02:54:20.883285 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62fd5040e324f7091a420da4d3ae67d" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:20.885985 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:20.886816 1657 raft_consensus.cc:2388] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 6: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:20.885397 1653 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 8.
I20250114 02:54:20.889914 1702 leader_election.cc:304] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:20.892275 1579 raft_consensus.cc:2463] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 8.
I20250114 02:54:20.895714 1702 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
W20250114 02:54:20.898224 3227 consensus_meta.cc:287] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454: Time spent flushing consensus metadata: real 0.540s user 0.008s sys 0.002s
I20250114 02:54:20.898643 3227 raft_consensus.cc:513] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.900442 3227 leader_election.cc:290] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:20.896221 3369 raft_consensus.cc:2743] T 2fccc76f52ed4204b3a8b03af5f65b1d P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 02:54:20.907982 3009 consensus_queue.cc:1035] T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 02:54:20.911111 1702 leader_election.cc:400] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625) with higher term. Message: Invalid argument: T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 6. Current term is 7.
I20250114 02:54:20.912739 1702 leader_election.cc:304] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:20.913504 2916 raft_consensus.cc:2743] T 464b2086919241de965a1ac17809451c P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:20.913874 1702 leader_election.cc:304] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:20.922103 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:20.922871 1662 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:20.925508 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:20.926295 1736 raft_consensus.cc:2363] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 6. Current term is 7.
I20250114 02:54:20.928825 3147 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 02:54:20.935142 3147 raft_consensus.cc:491] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1e5870f8269b49cf81dc39f1a5c8f5c1)
I20250114 02:54:20.935546 3147 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:20.938795 3075 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.938921 3302 raft_consensus.cc:3049] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 LEADER]: Stepping down as leader of term 7
I20250114 02:54:20.939298 3302 raft_consensus.cc:738] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:20.939328 3075 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.940112 3302 consensus_queue.cc:260] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.942618 3075 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:20.943212 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4346db2c8a3424ba9562e58d10084db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:20.943933 1729 raft_consensus.cc:2363] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 7. Current term is 8.
I20250114 02:54:20.935156 2916 raft_consensus.cc:2743] T 903edcc3b2294723af07e77c238bd3c8 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 02:54:20.945562 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30486d50dbd94c78a14bad5b98494276" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:20.946310 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11bad528717d449380b9e81d7cf948d3" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
W20250114 02:54:20.946415 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 71281us
I20250114 02:54:20.947002 1585 raft_consensus.cc:3054] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:20.947160 1553 leader_election.cc:304] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:20.948573 1662 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 7.
I20250114 02:54:20.953137 1702 leader_election.cc:400] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 5. Current term is 6.
I20250114 02:54:20.953521 1702 leader_election.cc:403] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250114 02:54:20.957793 1702 leader_election.cc:304] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:20.959187 2796 raft_consensus.cc:2798] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Leader election won for term 8
I20250114 02:54:20.959628 2796 raft_consensus.cc:695] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 8 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:20.960001 1702 leader_election.cc:304] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:20.960429 2796 consensus_queue.cc:237] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.964264 2868 raft_consensus.cc:2798] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 02:54:20.964728 2868 raft_consensus.cc:695] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 7 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:20.965466 2868 consensus_queue.cc:237] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:20.980476 3075 raft_consensus.cc:2743] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511) with higher term. Message: Invalid argument: T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 for earlier term 5. Current term is 6.
W20250114 02:54:20.993685 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 46587.8us
I20250114 02:54:20.994849 1553 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:21.008458 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:21.009282 1653 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 7.
I20250114 02:54:21.024667 1553 leader_election.cc:304] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:21.039597 1702 leader_election.cc:304] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:21.045631 1702 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:21.049497 3369 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 02:54:21.050001 3369 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:21.050415 3369 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Advancing to term 8
I20250114 02:54:20.979912 3176 raft_consensus.cc:513] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:21.033427 3303 raft_consensus.cc:2743] T 88a6663722394089be1417d1cf01653a P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 02:54:21.030210 3380 raft_consensus.cc:2743] T 5d77d39f9a4b4a57ae17db8e50237cdb P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:21.084872 3308 raft_consensus.cc:491] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:21.052109 3227 raft_consensus.cc:2743] T 30486d50dbd94c78a14bad5b98494276 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:21.087718 1733 raft_consensus.cc:2363] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 6. Current term is 7.
W20250114 02:54:21.088016 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 296637us
W20250114 02:54:21.088685 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 64141.7us
I20250114 02:54:21.047775 1585 raft_consensus.cc:2463] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 7.
I20250114 02:54:21.089354 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:21.107956 1553 leader_election.cc:304] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:21.109293 1587 raft_consensus.cc:1270] T f8434d11346144778e385ac18d417859 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20250114 02:54:21.101053 1472 catalog_manager.cc:4839] TS 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511): delete failed for tablet 985a08c846764478aaeb97aa5c283bf9 because the tablet was not found. No further retry: Not found: Tablet not found: 985a08c846764478aaeb97aa5c283bf9
I20250114 02:54:21.093286 3391 raft_consensus.cc:2743] T f37e0491ebc34f29b68a6425e22a3a2d P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 02:54:21.064515 3232 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:21.111760 1553 leader_election.cc:304] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:21.108203 3209 raft_consensus.cc:491] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
W20250114 02:54:21.212939 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.66:37511, user_credentials={real_user=slave}} blocked reactor thread for 160014us
W20250114 02:54:21.202610 3404 log.cc:926] Time spent T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20250114 02:54:21.085590 3308 raft_consensus.cc:513] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:21.213879 1584 raft_consensus.cc:1235] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250114 02:54:21.189582 2878 raft_consensus.cc:491] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
W20250114 02:54:21.217339 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 105655us
I20250114 02:54:21.219547 1553 leader_election.cc:400] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6.
I20250114 02:54:21.219934 1553 leader_election.cc:403] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250114 02:54:21.189167 3075 raft_consensus.cc:491] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:21.221097 1553 leader_election.cc:304] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
W20250114 02:54:21.223351 3397 log.cc:926] Time spent T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
W20250114 02:54:21.224737 3396 log.cc:926] Time spent T f83a9dac23e1497286938e63aa482439 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20250114 02:54:21.201469 3390 raft_consensus.cc:491] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b89550620e1d455d887be72232bb5454)
I20250114 02:54:21.225519 3390 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:21.227627 3390 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:21.227727 3308 leader_election.cc:290] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:21.214773 3209 raft_consensus.cc:513] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:21.232914 3209 leader_election.cc:290] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:21.228135 2878 raft_consensus.cc:513] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:21.198969 3338 raft_consensus.cc:2743] T a26544b19eeb4116a06caaba1da0e9d2 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 02:54:21.235981 3302 raft_consensus.cc:3054] T ff0a423b0ba64289acb34e5c4d31aad0 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Advancing to term 8
I20250114 02:54:21.236214 2878 leader_election.cc:290] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:21.228574 3075 raft_consensus.cc:513] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:21.237591 3338 raft_consensus.cc:2743] T 6b73d9b5d01245f8be76a02efe7e5eca P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:21.238762 1553 leader_election.cc:304] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:21.240445 3381 raft_consensus.cc:2743] T 4876e2bdbafa4d8684adc1348f1aeb80 P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 02:54:21.243546 1553 leader_election.cc:304] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:21.243537 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:21.245213 3075 leader_election.cc:290] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:21.246723 3380 raft_consensus.cc:2743] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 5. Current term is 6.
I20250114 02:54:21.247239 3381 raft_consensus.cc:2743] T 5e00ae680c2d4c5cbb5154723228286f P 62daa58e8ff64698bb46ea3fd5290cdd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:21.218492 3303 raft_consensus.cc:2743] T 7cd3f8af715b4ed8a97965a52b089a21 P 62daa58e8ff64698bb46ea3fd5290cdd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:21.254056 1553 leader_election.cc:304] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1
I20250114 02:54:21.256374 1553 leader_election.cc:400] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 6. Current term is 7.
I20250114 02:54:21.256651 1553 leader_election.cc:403] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250114 02:54:21.257676 1553 leader_election.cc:400] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 7. Current term is 8.
I20250114 02:54:21.257974 1553 leader_election.cc:403] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20250114 02:54:21.259076 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328b539725d54455aadcb0e76e17b4d6" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:21.284904 1553 leader_election.cc:304] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:21.287324 1553 leader_election.cc:400] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 6. Current term is 7.
I20250114 02:54:21.287730 1553 leader_election.cc:403] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250114 02:54:21.257493 3380 raft_consensus.cc:2798] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election won for term 7
I20250114 02:54:21.287818 3338 raft_consensus.cc:2743] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 7. Current term is 8.
I20250114 02:54:21.258725 2926 consensus_queue.cc:1035] T 9222bb8af9ab4f35a3fa735c302f11ac P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20250114 02:54:21.288092 3115 raft_consensus.cc:2743] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 6. Current term is 7.
I20250114 02:54:21.288827 3395 raft_consensus.cc:2743] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 02:54:21.312465 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:21.312570 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecc21b2d4ec4b0aab2d25073324044b" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:21.313980 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:21.315541 1734 raft_consensus.cc:1235] T ff0a423b0ba64289acb34e5c4d31aad0 P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Rejecting Update request from peer 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250114 02:54:21.315730 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:21.317577 1735 raft_consensus.cc:1270] T c62fd5040e324f7091a420da4d3ae67d P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:21.318826 1702 leader_election.cc:304] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:21.320349 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98adfd690b3746baa16ad8c1b99c83bd" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:21.320904 1653 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:21.303916 3391 raft_consensus.cc:3054] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:21.305245 3176 leader_election.cc:290] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:21.307740 3227 raft_consensus.cc:491] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 62daa58e8ff64698bb46ea3fd5290cdd)
I20250114 02:54:21.306918 3380 raft_consensus.cc:695] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 LEADER]: Becoming Leader. State: Replica: 62daa58e8ff64698bb46ea3fd5290cdd, State: Running, Role: LEADER
I20250114 02:54:21.335614 1653 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 7.
I20250114 02:54:21.323197 3009 raft_consensus.cc:2743] T 328b539725d54455aadcb0e76e17b4d6 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:21.343644 3147 raft_consensus.cc:513] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:21.345176 1728 raft_consensus.cc:1235] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Rejecting Update request from peer 1e5870f8269b49cf81dc39f1a5c8f5c1 for earlier term 6. Current term is 7. Ops: []
I20250114 02:54:21.325416 3232 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Requested vote from peers 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511), b89550620e1d455d887be72232bb5454 (127.0.106.67:42283)
I20250114 02:54:21.387535 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:21.388648 1653 raft_consensus.cc:2388] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd in current term 7: Already voted for candidate 1e5870f8269b49cf81dc39f1a5c8f5c1 in this term.
I20250114 02:54:21.345710 3227 raft_consensus.cc:513] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:21.399534 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:21.407158 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454" is_pre_election: true
I20250114 02:54:21.411326 1732 raft_consensus.cc:1270] T 1a1c92618e4c44119a063bfdaa75fc3e P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 02:54:21.412972 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a6663722394089be1417d1cf01653a" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:21.387133 3248 consensus_queue.cc:1035] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.015s
I20250114 02:54:21.424641 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ce953d409541bb85818e3dcf36541b" candidate_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:21.427160 1579 raft_consensus.cc:1270] T 9fec1da7d2f8404bb83e600bc99eb944 P 62daa58e8ff64698bb46ea3fd5290cdd [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:21.431066 1579 raft_consensus.cc:1270] T 07160adeb64840179d1c1529d86c94ed P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20250114 02:54:21.432211 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 112106us
I20250114 02:54:21.401472 3063 consensus_queue.cc:1046] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.032s
I20250114 02:54:21.453773 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:21.454702 1581 raft_consensus.cc:2388] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b89550620e1d455d887be72232bb5454 in current term 7: Already voted for candidate 62daa58e8ff64698bb46ea3fd5290cdd in this term.
I20250114 02:54:21.458504 2926 consensus_queue.cc:1035] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:21.394750 3380 consensus_queue.cc:237] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:21.461427 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
W20250114 02:54:21.462702 3403 log.cc:926] Time spent T 024e8460e81348268cc6fd408ece2a86 P 62daa58e8ff64698bb46ea3fd5290cdd: Append to log took a long time: real 0.175s user 0.001s sys 0.001s
I20250114 02:54:21.463331 1702 leader_election.cc:304] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:21.464740 3227 leader_election.cc:290] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:21.465286 3227 raft_consensus.cc:2743] T 0ecc21b2d4ec4b0aab2d25073324044b P b89550620e1d455d887be72232bb5454 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:54:21.466953 1702 leader_election.cc:304] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:21.462741 1662 raft_consensus.cc:1270] T c62fd5040e324f7091a420da4d3ae67d P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 02:54:21.423779 3369 raft_consensus.cc:513] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:21.471091 3063 raft_consensus.cc:3049] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 02:54:21.471576 3063 raft_consensus.cc:738] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1e5870f8269b49cf81dc39f1a5c8f5c1, State: Running, Role: LEADER
I20250114 02:54:21.472101 3369 leader_election.cc:290] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:21.472570 3063 consensus_queue.cc:260] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:21.474740 1730 raft_consensus.cc:2363] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 7. Current term is 8.
I20250114 02:54:21.435474 3147 leader_election.cc:290] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 election: Requested vote from peers 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65:46625), 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66:37511)
I20250114 02:54:21.482898 2868 raft_consensus.cc:2798] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 02:54:21.483382 2868 raft_consensus.cc:695] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 7 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:21.484187 2868 consensus_queue.cc:237] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:21.489022 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" is_pre_election: true
I20250114 02:54:21.491123 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "62daa58e8ff64698bb46ea3fd5290cdd"
I20250114 02:54:21.491639 1580 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 6 FOLLOWER]: Advancing to term 8
I20250114 02:54:21.492470 1553 leader_election.cc:400] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 7. Current term is 8.
I20250114 02:54:21.492761 1553 leader_election.cc:403] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20250114 02:54:21.493788 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:21.494343 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f63bdbf49400aae5fb92a633411db" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1"
I20250114 02:54:21.494933 1653 raft_consensus.cc:3054] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Advancing to term 8
@ 0x7fbce3393cf0 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
I20250114 02:54:21.498975 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7828a26fd7794b8a9ae9737e193416d2" candidate_uuid: "b89550620e1d455d887be72232bb5454" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" is_pre_election: true
I20250114 02:54:21.500197 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98ff91cae554f0f9347da0c886767ca" candidate_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b89550620e1d455d887be72232bb5454"
I20250114 02:54:21.474169 3063 raft_consensus.cc:3054] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Advancing to term 7
I20250114 02:54:21.502583 1580 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 8.
I20250114 02:54:21.503031 3273 raft_consensus.cc:2743] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 7. Current term is 8.
I20250114 02:54:21.521936 1580 raft_consensus.cc:1270] T f4346db2c8a3424ba9562e58d10084db P 62daa58e8ff64698bb46ea3fd5290cdd [term 8 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250114 02:54:21.529819 1653 raft_consensus.cc:2463] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 8.
I20250114 02:54:21.530907 3391 raft_consensus.cc:2743] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer b89550620e1d455d887be72232bb5454 (127.0.106.67:42283) with higher term. Message: Invalid argument: T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 62daa58e8ff64698bb46ea3fd5290cdd for earlier term 6. Current term is 7.
W20250114 02:54:21.531896 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 82855.9us
I20250114 02:54:21.538970 1585 raft_consensus.cc:2463] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 7.
I20250114 02:54:21.531121 3232 consensus_queue.cc:1035] T c62fd5040e324f7091a420da4d3ae67d P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.014s
W20250114 02:54:21.563972 1533 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250114 02:54:21.573365 1702 leader_election.cc:304] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454; no voters:
I20250114 02:54:21.581557 1735 raft_consensus.cc:1270] T 9fec1da7d2f8404bb83e600bc99eb944 P b89550620e1d455d887be72232bb5454 [term 5 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 02:54:21.598663 3209 consensus_queue.cc:1035] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250114 02:54:21.603809 1628 leader_election.cc:304] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:21.604936 1628 leader_election.cc:304] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1; no voters: 62daa58e8ff64698bb46ea3fd5290cdd, b89550620e1d455d887be72232bb5454
I20250114 02:54:21.607188 3233 raft_consensus.cc:2743] T 88a6663722394089be1417d1cf01653a P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 02:54:21.607365 3402 raft_consensus.cc:2743] T 01ce953d409541bb85818e3dcf36541b P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250114 02:54:21.608116 3402 consensus_queue.cc:1035] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:21.607959 3233 consensus_queue.cc:1035] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 02:54:21.611963 3369 raft_consensus.cc:2798] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 8 FOLLOWER]: Leader election won for term 8
I20250114 02:54:21.612468 3369 raft_consensus.cc:695] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [term 8 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:21.613510 3369 consensus_queue.cc:237] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
W20250114 02:54:21.628270 3411 log.cc:926] Time spent T 9222bb8af9ab4f35a3fa735c302f11ac P 1e5870f8269b49cf81dc39f1a5c8f5c1: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20250114 02:54:21.647536 1702 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 49548.1us
I20250114 02:54:21.649889 1553 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62daa58e8ff64698bb46ea3fd5290cdd; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454
I20250114 02:54:21.650569 1579 raft_consensus.cc:1270] T 11bad528717d449380b9e81d7cf948d3 P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 02:54:21.628085 3228 consensus_queue.cc:1035] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20250114 02:54:21.660987 1658 raft_consensus.cc:1270] T 11bad528717d449380b9e81d7cf948d3 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 02:54:21.661127 1659 raft_consensus.cc:1270] T f4346db2c8a3424ba9562e58d10084db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 8 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20250114 02:54:21.664554 1628 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.65:46625, user_credentials={real_user=slave}} blocked reactor thread for 58124us
I20250114 02:54:21.652390 3303 raft_consensus.cc:2743] T e98ff91cae554f0f9347da0c886767ca P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 02:54:21.686272 1736 raft_consensus.cc:1270] T 07160adeb64840179d1c1529d86c94ed P b89550620e1d455d887be72232bb5454 [term 6 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250114 02:54:21.690446 1657 raft_consensus.cc:2463] T e98ff91cae554f0f9347da0c886767ca P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b89550620e1d455d887be72232bb5454 in term 7.
I20250114 02:54:21.692456 1579 raft_consensus.cc:1270] T 1a1c92618e4c44119a063bfdaa75fc3e P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Refusing update from remote peer 1e5870f8269b49cf81dc39f1a5c8f5c1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 02:54:21.704948 3063 consensus_queue.cc:1035] T 9fec1da7d2f8404bb83e600bc99eb944 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:21.708511 1702 leader_election.cc:304] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b89550620e1d455d887be72232bb5454; no voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:21.718605 2916 raft_consensus.cc:2743] T 7828a26fd7794b8a9ae9737e193416d2 P b89550620e1d455d887be72232bb5454 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 02:54:21.727563 2688 consensus_queue.cc:1035] T 07160adeb64840179d1c1529d86c94ed P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250114 02:54:21.735917 2688 consensus_queue.cc:1035] T 1a1c92618e4c44119a063bfdaa75fc3e P 1e5870f8269b49cf81dc39f1a5c8f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
@ 0x7fbce3386dcd testing::UnitTest::Run() at ??:0
@ 0x37257c RUN_ALL_TESTS() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:?
@ 0x37143d main at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:?
I20250114 02:54:21.751863 2878 consensus_queue.cc:1035] T f4346db2c8a3424ba9562e58d10084db P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.012s
I20250114 02:54:21.754179 3228 consensus_queue.cc:1035] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.018s
W20250114 02:54:21.781674 3425 log.cc:926] Time spent T 024e8460e81348268cc6fd408ece2a86 P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20250114 02:54:21.782513 3176 consensus_queue.cc:1035] T 11bad528717d449380b9e81d7cf948d3 P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250114 02:54:21.786540 3421 log.cc:926] Time spent T 9222bb8af9ab4f35a3fa735c302f11ac P b89550620e1d455d887be72232bb5454: Append to log took a long time: real 0.103s user 0.000s sys 0.002s
I20250114 02:54:21.832474 1702 leader_election.cc:304] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5870f8269b49cf81dc39f1a5c8f5c1, b89550620e1d455d887be72232bb5454; no voters: 62daa58e8ff64698bb46ea3fd5290cdd
I20250114 02:54:21.857872 3227 raft_consensus.cc:2798] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 02:54:21.858405 3227 raft_consensus.cc:695] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [term 7 LEADER]: Becoming Leader. State: Replica: b89550620e1d455d887be72232bb5454, State: Running, Role: LEADER
I20250114 02:54:22.004091 1659 raft_consensus.cc:1270] T 98adfd690b3746baa16ad8c1b99c83bd P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20250114 02:54:22.006307 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 229726us
I20250114 02:54:21.938441 3227 consensus_queue.cc:237] T e98ff91cae554f0f9347da0c886767ca P b89550620e1d455d887be72232bb5454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } }
I20250114 02:54:22.014148 3215 consensus_queue.cc:1035] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.033s
I20250114 02:54:22.062618 1659 raft_consensus.cc:1270] T cab6c0814b8241c99bdeb347d765f929 P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 7 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 02:54:22.082299 3387 consensus_queue.cc:1035] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250114 02:54:22.149384 1578 raft_consensus.cc:1270] T 98adfd690b3746baa16ad8c1b99c83bd P 62daa58e8ff64698bb46ea3fd5290cdd [term 7 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250114 02:54:22.152910 2878 consensus_queue.cc:1035] T 98adfd690b3746baa16ad8c1b99c83bd P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250114 02:54:22.202035 1553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.67:42283, user_credentials={real_user=slave}} blocked reactor thread for 73078.4us
I20250114 02:54:22.209800 1729 raft_consensus.cc:1270] T cab6c0814b8241c99bdeb347d765f929 P b89550620e1d455d887be72232bb5454 [term 7 FOLLOWER]: Refusing update from remote peer 62daa58e8ff64698bb46ea3fd5290cdd: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250114 02:54:22.219007 3380 consensus_queue.cc:1035] T cab6c0814b8241c99bdeb347d765f929 P 62daa58e8ff64698bb46ea3fd5290cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 02:54:22.227448 1585 raft_consensus.cc:1270] T 4d6f63bdbf49400aae5fb92a633411db P 62daa58e8ff64698bb46ea3fd5290cdd [term 8 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250114 02:54:22.228578 3176 consensus_queue.cc:1035] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 02:54:22.257906 1483 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39748 (request call id 101) took 4695 ms. Trace:
I20250114 02:54:22.258740 1482 catalog_manager.cc:5526] T 024e8460e81348268cc6fd408ece2a86 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 2 to 4, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to b89550620e1d455d887be72232bb5454 (127.0.106.67). New cstate: current_term: 4 leader_uuid: "b89550620e1d455d887be72232bb5454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:22.266688 1482 catalog_manager.cc:5526] T f8434d11346144778e385ac18d417859 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:22.268805 1482 catalog_manager.cc:5526] T ff0a423b0ba64289acb34e5c4d31aad0 P 1e5870f8269b49cf81dc39f1a5c8f5c1 reported cstate change: term changed from 3 to 7, leader changed from 1e5870f8269b49cf81dc39f1a5c8f5c1 (127.0.106.66) to 62daa58e8ff64698bb46ea3fd5290cdd (127.0.106.65). New cstate: current_term: 7 leader_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62daa58e8ff64698bb46ea3fd5290cdd" member_type: VOTER last_known_addr { host: "127.0.106.65" port: 46625 } } peers { permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 } } peers { permanent_uuid: "b89550620e1d455d887be72232bb5454" member_type: VOTER last_known_addr { host: "127.0.106.67" port: 42283 } } }
I20250114 02:54:22.285805 1653 raft_consensus.cc:1270] T 4d6f63bdbf49400aae5fb92a633411db P 1e5870f8269b49cf81dc39f1a5c8f5c1 [term 8 FOLLOWER]: Refusing update from remote peer b89550620e1d455d887be72232bb5454: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250114 02:54:22.287068 2878 consensus_queue.cc:1035] T 4d6f63bdbf49400aae5fb92a633411db P b89550620e1d455d887be72232bb5454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5870f8269b49cf81dc39f1a5c8f5c1" member_type: VOTER last_known_addr { host: "127.0.106.66" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
@ 0x7fbcdfa0fbf7 __libc_start_main at ??:0
@ 0x293fca _start at ??:?